Skip to content

fix(ds4): propagate sampler config in layer split#508

Draft
Graffioh wants to merge 1 commit into
Luce-Org:codex/ds4-rocmfpx-serverfrom
Graffioh:codex/fix-ds4-layer-split-sampling
Draft

fix(ds4): propagate sampler config in layer split#508
Graffioh wants to merge 1 commit into
Luce-Org:codex/ds4-rocmfpx-serverfrom
Graffioh:codex/fix-ds4-layer-split-sampling

Conversation

@Graffioh

@Graffioh Graffioh commented Jul 11, 2026

Copy link
Copy Markdown
Contributor

Summary

  • propagate the request sampler configuration into DeepSeek4LayerSplitAdapter
  • preserve request temperature, top-p, top-k, repetition/frequency/presence penalties, and seed instead of silently retaining the adapter defaults
  • reseed the adapter RNG when deterministic sampling is explicitly requested
  • add unit coverage for sampler propagation and deterministic reseeding

Stack

Stacked on #503. Base is codex/ds4-rocmfpx-server, so this PR contains only the DeepSeek4 layer-split sampler propagation fix. Retarget to the branch where #503 lands once it is merged.

Problem

LayerSplitBackend::run_from_state() calls adapter_->begin_request(req) before prefill and decode. The Qwen35, Gemma4, and Laguna layer-split adapters copy req.sampler into their request-local sampler state, but the DeepSeek4 adapter discarded the request:

void DeepSeek4LayerSplitAdapter::begin_request(const GenerateRequest & req) {
    (void)req;
}

As a result, DeepSeek4 layer-split requests silently retained the default sampler configuration. A request sent with temperature: 0.01 and seed: 42 still reached run_layer_split_ar_decode() with sampler.temp == 0 under GDB.

The adapter now follows the same contract as the other layer-split implementations:

sampler_ = req.sampler;
if (req.do_sample && sampler_.seed != 0) {
    sampler_rng_.seed(sampler_.seed);
}

A non-zero seed explicitly requests reproducible sampling. Seed zero keeps the RNG's existing nondeterministic state and does not disable temperature sampling.

Validation

  • branch rebased directly onto the current fix/feat(ds4): ROCmFPX byte-safe decode + fused-decode path + ds4 optimizations #503 head (6c7ee19)
  • git diff --check: passed
  • diff is limited to the DeepSeek4 layer-split adapter and its unit test
  • pre-fix GDB validation: an HTTP request with temperature: 0.01 reached run_layer_split_ar_decode() with sampler.temp == 0
  • post-fix Release/RelWithDebInfo CUDA H200 build: passed
  • post-fix request with temperature: 0.01 and seed: 42 returned the coherent response The capital of France is Paris.
  • unit coverage verifies temperature/top-p/seed propagation and deterministic RNG reseeding

Follow-up

TO INVESTIGATE: Greedy layer-split decode with temperature: 0 has a separate common-runtime issue: run_layer_split_ar_decode() currently skips logits processing instead of selecting their argmax. That cross-model fix is intentionally kept out of this DeepSeek4-specific change.

Review in cubic

@Graffioh Graffioh changed the title fix(deepseek4): propagate sampler config in layer split fix(ds4): propagate sampler config in layer split Jul 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant