feat(ds4): add indexed layer-major prefill#520
Draft
davide221 wants to merge 2 commits into
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
exact,dense, andsparseDeepSeek V4 prefill modes;exactremains the defaultThe optimized path is explicit (
--ds4-prefill sparse). It does not add runtime environment-variable switches or change the default prefill policy.Performance
DeepSeek-V4-Flash-ROCMFP2-STRIX, gfx1151, ROCm 7.2.2,
--chunk 2048 --max-ctx 32768, all experts resident, model top-k 6, no draft, and all prefix/prefill caches disabled.Pinned ranges were 314.9–315.2 tok/s at 2K, 274.1–277.8 at 8K, and 236.4–238.6 at 24K. The first cold 2K request was 285.8 tok/s. The clock pin on the test host was:
This is 300+ tok/s for a warm 2K prompt, not a sustained 300+ tok/s claim at longer context.
Correctness boundary
exactkeeps tokenwise scheduling and does not enter the layer-major path.exactrequests produced the same token sequence in this branch. The current fix/feat(ds4): ROCmFPX byte-safe decode + fused-decode path + ds4 AR optimizations #503 baseline varied across identical repeated requests, so this PR does not claim equality to one nondeterministic baseline capture.sparseis experimental and may change logits or full generated sequences. It needs task-level quality validation before it can become a default.Validation
test_deepseek4_unitgit diff --check