Skip to content

feat: dynamically split PD decode requests - #1535

Open
shihaobai wants to merge 15 commits into
mainfrom
bsh/dynamic_pd_chunk
Open

feat: dynamically split PD decode requests#1535
shihaobai wants to merge 15 commits into
mainfrom
bsh/dynamic_pd_chunk

Conversation

@shihaobai

Copy link
Copy Markdown
Collaborator

Summary

  • replace the static LIGHTLLM_PD_SPLIT_MAX_NEW_TOKENS chunking loop with capacity-driven PD segments
  • use aggressive EMA-based admission for PD Decode without applying the router-side batch_max_tokens prefill gate
  • when an overlapped D request runs out of KV capacity, lower its max_new_tokens to the current accepted output length so the existing post-handle path naturally returns finish_reason=length
  • continue the logical request on the same P/D pair so incremental prompts can reuse cache; non-overlap Decode keeps the existing pause/recover behavior

Tests

  • 16 passed: PD dynamic split, PD master cached-token handling, and PD selector tests
  • Python compile checks for the modified router/backend modules
  • git diff --check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants