Skip to content

feat(ds4): add indexed layer-major prefill#520

Draft
davide221 wants to merge 2 commits into
codex/ds4-rocmfpx-serverfrom
feat/ds4-indexed-prefill
Draft

feat(ds4): add indexed layer-major prefill#520
davide221 wants to merge 2 commits into
codex/ds4-rocmfpx-serverfrom
feat/ds4-indexed-prefill

Conversation

@davide221

@davide221 davide221 commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add typed exact, dense, and sparse DeepSeek V4 prefill modes; exact remains the default
  • execute optimized prefill layer-major with batched causal compressor updates and device-resident HC/FFN state
  • add the DS4 learned indexer QAT/score/mask kernels and the D=512 compact sparse-attention path
  • add gfx1151 ROCmFPX MMQ support, shared activation quantization, and grouped output projection without materialized permutations

The optimized path is explicit (--ds4-prefill sparse). It does not add runtime environment-variable switches or change the default prefill policy.

Performance

DeepSeek-V4-Flash-ROCMFP2-STRIX, gfx1151, ROCm 7.2.2, --chunk 2048 --max-ctx 32768, all experts resident, model top-k 6, no draft, and all prefix/prefill caches disabled.

Prompt Tokens Auto clock Pinned 2.9 GHz
2K 2,131 287.0 tok/s warm 315.0 tok/s warm
8K 7,641 258.0 tok/s 274.9 tok/s median
24K 22,841 225.8 tok/s 237.0 tok/s median

Pinned ranges were 314.9–315.2 tok/s at 2K, 274.1–277.8 at 8K, and 236.4–238.6 at 24K. The first cold 2K request was 285.8 tok/s. The clock pin on the test host was:

echo high | sudo tee /sys/class/drm/card2/device/power_dpm_force_performance_level

This is 300+ tok/s for a warm 2K prompt, not a sustained 300+ tok/s claim at longer context.

Correctness boundary

  • exact keeps tokenwise scheduling and does not enter the layer-major path.
  • Three identical exact requests produced the same token sequence in this branch. The current fix/feat(ds4): ROCmFPX byte-safe decode + fused-decode path + ds4 AR optimizations #503 baseline varied across identical repeated requests, so this PR does not claim equality to one nondeterministic baseline capture.
  • Sparse 2K/8K/24K one-token gates produced the same token id and SHA-256 across every run.
  • sparse is experimental and may change logits or full generated sequences. It needs task-level quality validation before it can become a default.

Validation

  • full ROCm/gfx1151 build
  • test_deepseek4_unit
  • deterministic exact-mode repeat capture
  • sparse 2K/8K/24K sequence and throughput captures
  • git diff --check

Review in cubic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant