Skip to content

cuda: fix an experts-rows test race and measure the ModelOpt checkpoint on one GB10 - #1

Merged
akol1 merged 2 commits into
akol1:codex/flashnext-cuda-nativefrom
balancedscorpion:fp4local-qualify
Oct 18, 2026
Merged

akol1 merged 2 commits into
akol1:codex/flashnext-cuda-nativefrom
balancedscorpion:fp4local-qualify

Conversation

@balancedscorpion

Copy link
Copy Markdown

For ashhart#952 (ashhart#952). Two commits on top of beec1aad6, independent of each other, so either can be
taken alone.

What this changes

  • test(cuda): clear experts-rows buffers on the kernels' stream. Rig.zeros in
    zig/tests/cuda/experts_rows_tests.zig cleared each buffer with fill8(0, null), which is cuMemsetD8 on the
    default stream. The kernels run on a non-blocking stream that does not wait for it, so on sm_121 the clear of the
    one-row reference's output sometimes landed after the decode kernel wrote it, and tf-cuda-test experts-rows
    failed: FAIL experts_rows SwiGLU rows: first difference at byte 0, 76800 of 76800 words differ. The change clears
    on the rows' stream. One line, test only.
  • docs: measure the ModelOpt FP4-Local checkpoint on one GB10. A subsection, "The ModelOpt checkpoint on one
    GB10", in docs/recipes/flash-next-cuda.md: decode, concurrency, cold first tokens to 131,072 tokens, the
    exactness and seam checks, and what a 128 GB machine admits. All measured with beec1aad6 built unpatched. Docs
    only.

Receipt

  • Environment:
    • TensorFold beec1aad6c967f6e32aa14a3fd21ca8358710279, built with -Dnvcc -Dsm=121 -Doptimize=fast, nvcc 13.3,
      zig 0.17.0, served with TENSORFOLD_CUDA_KERNELS=native.
    • One NVIDIA GB10 (sm_121, 128 GB unified), driver 580.178.04, Ubuntu 24.04.5 aarch64.
    • nvidia/Qwen3.8-Flash-Next-FP4-Local revision b9ce18a6887e18c089fdea8509a33e63429a3300. No MLX or PyTorch.
  • Exactness, served code unchanged by this PR (cuda: add experimental native Flash Next family and local tensor parallelism ashhart/TensorFold#952 as shipped, greedy, --prompt-cache-gib 0):
    • bench_concurrent.py --alone --serial --levels 1,4,8 --tokens 256 --reps 3, three runs: concurrent equals solo
      78/78 and drafted equals "draft": false 6/6 in each.
    • A cancelled reply, and a reply after a restart, equal the fresh reply's token_sha256. A second conversation
      turn resumed from the prompt cache (--no-drafts) equals the same turn computed without it.
    • flashnext-fidelity-single against -serial at 7 fixed histories of 20 to 6,496 tokens: 7/7 byte-equal.
  • Decode speed before and after: n/a, no served code changes. The recipe's numbers are cuda: add experimental native Flash Next family and local tensor parallelism ashhart/TensorFold#952 as shipped: 64.1-64.6 /
    40.7-41.1 tok/s (fibonacci-raw / gpu-chat-no-think, one stream).
  • Prompt speed before and after: n/a, no change reaches prompt processing. The recipe's cold first tokens at
    2k / 8k / 32k / 64k / 128k: 1.83 / 7.11 / 29.2 / 60.1 / 131.7 s (bench_cache.py --require-cold, exact arrays).
  • Tests run:
    • tf-cuda-test experts-rows, thirty runs each, alternated, same build flags: before 8/30 failed, after 0/30.
    • tf-cuda-test kernel-contract on sm_121: SUMMARY 32 kernels: 32 pass gates 1-4, 0 fail, 0 known findings; 5 controls: 5 red as planted, 0 missed; 0 limit problems.
    • CI's No GPU job: zig build test 104/104 steps, 1,483/1,490 passed, 7 skipped (libjpeg-turbo absent);
      test-kernel-contract, test-golden and fuzz-cortheon pass.
    • test-cuda-host test-flashnext-contract-host for sm_121: 204/204.
    • lean_check.py: 0 new. zig fmt --check build.zig zig: clean.
  • Not run, and why:
    • Metal, SM 80/86/89/120 and tensor parallelism: no hardware.
    • Six other test files pair fill8(0, null) with non-blocking streams (exlr8_tests, bench, oracle_tests,
      exlr8_dense_tests, libs_tests, runtime_tests). I have not seen them fail and did not change or check them.
    • A 262,144-token cold prompt: the window fitted a stream in one start of four here.

Checklist

  • Every token still goes through the lane rounds. No serial path, and nothing that needs drafts off. (No served
    code changes.)
  • Drafted output equals "draft": false, a resumed prompt equals a fresh one, and concurrent equals solo.
  • No precision traded for speed. No bits change.
  • A new or changed GPU kernel is registered with the kernel contract. (No kernel changes; the sm_121 SUMMARY
    line is above.)
  • Prompt processing is no slower than the last release. (Untouched.)
  • The description says which platforms I ran, Metal M1 to M5 and CUDA, and which I could not.
  • New tests fail before the change, pass after it, and skip cleanly without their dependency. (An existing GPU
    test: 8/30 failed before, 0/30 after.)
  • Comments and docstrings are one line. No measurements or history in the source.
  • No personal data, machine names, internal hosts or local paths. No AI attribution lines.

Rig.zeros cleared each buffer with cuMemsetD8 on the default stream, while the kernels run on a non-blocking
stream that does not wait for it. On a GB10 (sm_121) the clear of the one-row reference's output sometimes
landed after the decode kernel wrote it, and experts-rows failed with an all-zero or partly zero reference:
8 of 30 runs at beec1aa, 0 of 30 with this change, alternated on one machine. Clear on the rows' stream.
The recipe's measurements covered only the MLX affine checkpoint. Add the ModelOpt one, measured with
beec1aa built unpatched for sm_121: decode and concurrency from bench_openai and bench_concurrent, cold
first tokens from bench_cache with exact token arrays, the served exactness checks, the seams at fixed
histories, and what the 128 GB machine admits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants