Repository navigation
Conversation
…wer: batched prompt passes (prefill_many, addStreams), a backend's refusals for a memory guard, follow() for a tensor-parallel rank 1, shared-pool admission fields, image and video inputs for the native server (vision_inputs, media_fetch, qwen_image, stb_image)
…cel; lean rule and zig fmt setup() returned cancel()'s true for a background warm pass that yields to a newer request, so the caller launched the job cancel() had just finished (SEGV in lane_host_warm_test). It now returns false, as for a drop. Comments one line each; lean_check skips zig/vendor/ (stb_image.h, as on ashhart#726). lane_host.zig and lanes/engine.zig stay under 600 lines: the batched prompt pass moves to lane_host_admit.zig (many, one) and lanes/open.zig (addStreams, adoptPrefilled), aliased where they were.
…rozen words (the errors/content-image golden)
…nsorfold-native takes memcpy from libSystem again (zig build test on macOS)
…eports; images wait for the shared decoder; no refused hook - prompt_cache: Entry.pins, Store.pin/unpin. victimBut passes pinned entries over, so a later job's lookup in admit.many (or a keep during the batched pass) no longer frees a state an earlier job has yet to restore; a keep with nothing evictable fails instead of unwrapping null. LaneHost.setup pins, reported and requeue unpin. - vision: zig/vendor/stb dropped and no longer compiled. qwen_image.decode refuses by name (NoDecoder) until the shared runtime decoder (pinned libjpeg-turbo) lands; lean_check's vendor exemption goes with it. - lanes: Backend.VTable.refused, Engine.refused, cuda_bound's forwarding and the fake's stray refusedFn removed; nothing consumed them.
…roup-32 token, so the CUDA replay starts from the row the Python engine reads.
… so the linear-attention input is that mixed row.
…f the mixed row on one token, so the DeltaNet input is that projection.
…y match the host chain on one token, all 6144 gated values and all 2560 projected values, so the mixer branch is that row.
…, the write-back exact and its sums within 1 ulp, one normed value 1 step off, and the mixed row within 2 steps, so the expert input is that row.
…he shared expert exact, router logits within 29184 ulps, two weights 1 step off, the activations exact, and every down value the same bf16 with fp32 within 16384 ulps.
…the projection, the prepared heads, the one-key output, the gate and the output row all exact.
…one projected value 1 step off, and the prepared heads, the one-key output, the gate and the output row all exact.
… one projected value 1 step off, and the prepared heads, the one-key output, the gate and the output row all exact.
… the projection, the prepared heads, the one-key output, the gate and the output row all exact.
… the projection, the prepared heads, the one-key output, the gate and the output row all exact.
… the projection, the prepared heads, the one-key output, the gate and the output row all exact.
… two projected values 1 step off, one gate value 1 step off with group sums within 1024 ulps, and the prepared heads, the one-key output and the output row all exact.
… one gate value 1 step off with group sums within 8192 ulps, and the projection, the prepared heads, the one-key output and the output row all exact.
… one prepared key value 1 step off, and the projection, the other prepared heads, the one-key output, the gate and the output row all exact.
… one projected value 1 step off, and the prepared heads, the one-key output, the gate and the output row all exact.
… one projected value 1 step off, and the prepared heads, the one-key output, the gate and the output row all exact.
… two projected values 1 step off, and the prepared heads, the one-key output, the gate and the output row all exact.
…all 2560 values and the group sums exact.
…token, all 2560 mixed values exact, and three of 248320 scores 1 step off.
…ing every expert onto the GPU once.
…m; prompts past 512 keys; LR7 capture. The ids and position uploads were synchronous copies, which do not wait for the non-blocking compute stream, so a later row overwrote a queued row's token id or position: wrong tokens on most prompts. Both now go on the stream. attnStep launches one 512-key chunk per started chunk of the row's keys (as attention.attention does) instead of always one, so prompts run to the captured NCH * 512 keys; the lanes engine defaults to that window and refuses a larger one. The prompt check prints each run's top five scores and its timing. zig/tests/cuda/flashnext: capture.py and capture.sh (Python engine 0.6.1, drafts off, greedy, 1536 cache slots) and the Unbound-LR7 prompts with their references; flashnext-prompt matches the teacher paths on all three.
…ules and device weight bytes from what it declares; Flash Next registers The registry gains flashnext.native. What was Nemotron's by name is now read from the family's declarations: its prompt-cache copies (snap_bytes, snap_save, snap_restore, snap_drop, and snap_ready's spare buffers), the store's rules (cache_rules; without them a state keeps at any row with the MTP head's one token on, ashhart#670), the quiet wait before a warm pass and the lone driver's hand-over budget. Nemotron declares the copies, the wait and the budget, so its paths are unchanged; a family without copies gets no prompt cache, and the startup line says so. A family that keeps part of its checkpoint off the device says what it uploads (deviceWeightBytes) and admission counts that, not every shard: Flash Next leaves its n-gram tables mapped and loads no vision tower, so the MLX 4-bit checkpoint uploads 73.3 of its 105.4 GiB, where counting every shard made the default reserve on a GB10 refuse it by 0.1 GiB. --slide is refused before anything loads for a family that declares no learner. The Flash Next family names what the host reads: one segment, its prompt rows, each stream's bytes, its refusals in words, and the native host's lone-driver signature.
…ashnext-review-integration
|
Thank you, @akol1, the maintained integration and affine/ModelOpt stages make #952 the Flash Next CUDA base for us to finish. We checked beec1aa. cuda_lanes.zig:237-244 silently clears drafting on the fourth active lane; we will remove that fallback with correctly sized/priced state and a four-plus drafted-lane regression. TP draft snapshots also need implementation or a clear startup refusal for the unsupported cache combination. We will carry @ZackSample's #869 tests/fixtures with credit, qualify affine single-rank first, then ModelOpt, then TP. The named --vision refusal stays until owned images are implemented. Please retain the teacher/reference and state-seam fixtures for fresh device gates, including actual draft activity, cache equality and NVIDIA default math. Posted by Ash's AI Assistant Synthena |
cwschroeder
left a comment
cwschroeder
left a comment
There was a problem hiding this comment.
Correction to my earlier review and its linked CPU runner: the old --mode fast command placed -Ofast after all module declarations, leaving the root in Debug. That invocation did not prove Fast coverage. The corrected runner applies the mode before every -M and asserts the root mode at compile time in its temporary export.
I reran the original pinned candidate with that correction and independently checked the adopted guard at beec1aad: all four behavior cases pass in Debug and Fast. Removing only the two guards in a temporary copy makes both drafted cases fail in each mode; both plain restore controls still pass. These exercise actual lane open and explicit restore with virtual CUDA callbacks and a nonblocking local control peer. No CUDA/NCCL library or device is opened.
Astra found no material regression in the adoption delta. The guard and test file retain the offered patch and authorship, so my bounded restore finding is closed. Whole-tree format and whitespace pass; lean is 517/517 against main 6ee828c2, zero new. All four current-head GitHub checks are green. This leaves Ash's fourth-lane drafting and unsupported-cache startup work with the existing owner, and adds no TP+MTP snapshot support, model-parity or performance qualification.
|
Thanks for putting this together. I've been running this branch on a DGX Spark (GB10, sm_121) with NVIDIA's ModelOpt checkpoint,
- try b.fill8(0, null);
+ // on the kernels' stream: it is non-blocking, so a memset on the default stream may land after them
+ try b.fill8(0, r.s.handle);Thirty runs each, alternated with the same build flags: 8/30 failed before, 0/30 after. Six other test files pair While I had it running I checked the rest of the ModelOpt path on this machine, and everything else held up: concurrent equals solo, drafted equals Measurements: the ModelOpt checkpoint on one GB10 at
|
Rig.zeros cleared each buffer with cuMemsetD8 on the default stream, while the kernels run on a non-blocking stream that does not wait for it. On a GB10 (sm_121) the clear of the one-row reference's output sometimes landed after the decode kernel wrote it, and experts-rows failed with an all-zero or partly zero reference: 8 of 30 runs at beec1aa, 0 of 30 with this change, alternated on one machine. Clear on the rows' stream.
The recipe's measurements covered only the MLX affine checkpoint. Add the ModelOpt one, measured with beec1aa built unpatched for sm_121: decode and concurrency from bench_openai and bench_concurrent, cold first tokens from bench_cache with exact token arrays, the served exactness checks, the seams at fixed histories, and what the 128 GB machine admits.
|
@balancedscorpion both commits from your branch are on this head with your authorship — The The recipe section went in as measured on your GB10, with the checkpoint revision, driver and window settings attached. Six other test files pair New head |
…eckpoints, with MTP drafts and tensor ranks The family registry lists Flash Next beside Qwen. The merge keeps the existing cuda_window.zig and drops #952's copy of it, cuda_context.zig. Opened carries both the drain deadline and #952's follower hook, and the kernel list, the experts pack instantiations and the build's module and check lists take both sides.
…d a drain that fails before it succeeds #952 already carries #869's staging fix. Its regression now also fails the first copy beside the second copy and the record, treats a failed copy as possibly queued, tells an event wait from the stream drain it requires, and adds a failed drain then a retry. The guard stays set until the drain succeeds. The production staging code is unchanged. Co-authored-by: ZackSample <134018356+ZackSample@users.noreply.github.com>
…ions.zig, so cuda.zig stays under 600 lines #952's rank and follower wiring took native/cuda.zig to 611 lines. cuda_options.zig now sets the ranks, vision, --precision and fp8-cache options a family takes, unchanged, and openWith calls it. cuda.zig is 593 lines. The staging test's title now says it covers a failed first upload too.
… nvcc 13.2's in its own set The release fatbins are nvcc 13.0's, and #952 recorded its launches as nvcc 13.2 compiles them. For the release compiler 29 sm_120 and sm_121 entries of 17 launches were stale, and the live check on a GB10 failed 15 of them. Those records now hold 13.0's usage, read from every SM's fatbins on a GB10, and the 13.2 set holds each launch's 13.2 usage, read the same way on an RTX PRO 6000. sm_80, sm_86 and sm_89 were already right.
…ared Qwen-VL tower, and our follow-ups
|
Thank you, @akol1, @ZackSample, @AdrianBinDC, @Bizuayeu and @cwschroeder, for Flash Next CUDA, the staging fix and its tests. We merged #952 for 1.1.0 in 53abe45, retaining #869's credit. Drafts stay enabled for every admitted stream. Tensor-rank drafting with a prompt cache is refused early and names The reported GB10 MLX 4-bit run with its MTP head passes 17 exactness cells, two/four concurrent comparisons and 12 cached chat turns. PNG checks pass; JPEG checks still need the decoder. Launch records distinguish nvcc 13.0 and 13.2. NVIDIA GB10 affine-4 MTP passed 17/17, concurrent equality and chat 12/12; image warm-turn/JPEG gaps and the NVFP4 pair remain limited. Posted by Ash's AI Assistant Synthena |
8 participants
Native Flash Next CUDA family on the maintained engine: ModelOpt NVFP4/CSF expert weights, MXFP8 and BF16 dense projections, GDN/attention/HC operators, two-rank placement, draft verification and family-scoped tests. The shared scheduler, Recall, admission, health and HTTP contracts remain in the core.
This includes the bounded dependencies from #904, #906, #907, #908, #912, #929 and #945, preserving their commits and the original Flash Next contributors. Related discussion: #472; prior contributor work: #476 and #869.
Source:
beec1aad6c967f6e32aa14a3fd21ca8358710279; current main base:b090e3b19014bfa1eb2b7f38ea3cc4d39abec9be.Review fixes:
NoHeadStatebefore sending a restore frame or copying history. Plain lanes retain their restore behavior. This does not add MTP snapshot support; positive-cache TP+MTP remains excluded by the recipe.Final-source local validation, Zig 0.17.0, macOS 26.6.2 arm64:
zig build native test test-golden test-kernel-contract fuzz-cortheon -Dcpu=apple_m1 -Dversion=1.0.0 -j2: 236/236 steps succeeded; 2,740 passed and four skipped tests among executed artifacts, with unchanged build artifacts reused from cache. The standard workflow includes existing Metal unit tests; the new restore/checker/dispatch regressions are CPU-only.Cuda.open/snapRestorecases pass in Debug and Fast. Before the guard, both drafted cases reproduce the defect while both plain controls pass. Removing destination poison or the finite-error guard makes the projection controls fail.Contributor follow-up on this head, both commits by @balancedscorpion from
akol1/TensorFold#1, without a merge commit:ab7f7baatest(cuda): clear experts-rows buffers on the kernels' stream— test only.Rig.zeroscleared each buffer withfill8(0, null), which iscuMemsetD8on the default stream, while the row kernels run on the rig's non-blocking stream; on sm_121 the clear could land after the decode kernel wrote the one-row reference output. Clearing on the rig stream removes the overlap. His receipt: thirty alternating runs oftf-cuda-test experts-rowson the same build, 8/30 failed before and 0/30 after.b6fd0d92docs: measure the ModelOpt FP4-Local checkpoint on one GB10— docs only, the new "The ModelOpt checkpoint on one GB10" section:nvidia/Qwen3.8-Flash-Next-FP4-Localatb9ce18a6, three runs of the defaults, concurrent equal to solo and drafted equal to"draft": false, cold first tokens to 131,072, and the admission numbers for a 128 GB machine.Neither commit changes served code, so the local validation above remains the receipt for the served source. Six other test files also pair
fill8(0, null)with a non-blocking stream; they are unchanged here and none has shown the failure.Historical device evidence, bound to
46836015d7d8fcdab4882aea8075fa69c996d05b:flashnext-hc,flashnext-attn,route-rows,norm-rows,experts-rows,flashnext-draft-affine,flashnext-shared,flashnext-nvfp4-tp, andflashnext-dense-hc. HC modes 2 and 4 each compared 17,408 independent FP64 columns: zero worse and 1,053 improved.Remaining qualification gates:
--prompt-cache-gib 0. Text profiles only. SM 80/86/89/121 hardware and Metal family execution are unrun.This remains a draft. GitHub CI on
b6fd0d92is running; the green desktop, mobile, Zig format and No GPU host/contract/golden/fuzz/lean run cited above was forbeec1aad6. Compiler and kernel success do not qualify a checkpoint or the complete serving runtime.