[https://nvbugs/6284101][fix] Fix guided decoding for Harmony parser - #15741
[https://nvbugs/6284101][fix] Fix guided decoding for Harmony parser#15741dongfengy wants to merge 11 commits into
Conversation
558e70c to
ab42d37
Compare
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughAdds reasoning-parser-aware guided-decoding adaptation for Harmony and other reasoning formats. Wires the shared adaptation into LLM sampling and OpenAI serving. Adds final-content extraction and reasoning-aware JSON evaluation. ChangesReasoning-parser guided decoding and evaluation
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Client
participant LLM
participant ReasoningParser
participant JsonModeEval
Client->>LLM: Submit guided-decoding parameters
LLM->>ReasoningParser: Resolve and adapt parser framing
ReasoningParser-->>LLM: Return adapted guided-decoding parameters
LLM-->>JsonModeEval: Return generated output
JsonModeEval->>ReasoningParser: Extract final content
ReasoningParser-->>JsonModeEval: Return parsed content
Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/evaluate/interface.py`:
- Around line 90-109: The fallback parsing in interface.py is catching all
Exception in both the harmony adapter path and the ReasoningParserFactory path,
which can mask real bugs. Update the try/except blocks around
get_harmony_adapter().harmony_output_to_openai and parser.parse(text) to catch
only the expected import, parsing, or type-related failures, and leave
unexpected exceptions to surface. Keep the same fallback-to-text behavior, but
narrow the exception handling in these parser paths to the smallest specific set
possible.
In `@tensorrt_llm/evaluate/json_mode_eval.py`:
- Around line 33-48: Add a return type annotation to _load_json_from_generation
in json_mode_eval.py, and if helpful introduce a local alias for the JSON value
type to keep the signature readable. Update the function signature of
_load_json_from_generation (and any related helper typing if needed) so the new
helper is fully annotated per the codebase guidelines.
In `@tests/unittest/evaluate/test_generation_content.py`:
- Around line 1-2: The new test file currently has only an SPDX-only header, but
it needs the repository’s full NVIDIA copyright/license header block. Update the
top-of-file header in test_generation_content.py to match the required NVIDIA
format used elsewhere in the repo, including the year of the latest meaningful
modification and the standard Apache-2.0 license notice.
- Around line 23-42: Add a test in test_generation_content for
extract_final_content_from_generation that combines token_ids with
reasoning_parser="qwen3" and asserts the qwen3 path still wins. Mirror the
existing test_extract_final_content_does_not_guess_harmony_from_tokens setup by
monkeypatching tensorrt_llm.serve.harmony_adapter.get_harmony_adapter to raise
if called, then verify the function returns the expected final content. This
should cover the explicit non-Harmony parser branch even when token_ids are
present and prevent future regressions in extract_final_content_from_generation.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 318d2a29-586e-453d-ab59-726c7b606634
📒 Files selected for processing (7)
tensorrt_llm/evaluate/interface.pytensorrt_llm/evaluate/json_mode_eval.pytensorrt_llm/llmapi/llm.pytensorrt_llm/llmapi/reasoning_parser.pytensorrt_llm/serve/openai_protocol.pytests/unittest/evaluate/test_generation_content.pytests/unittest/llmapi/test_sampling_params.py
|
/bot run --disable-fail-fast |
ab42d37 to
059d1c9
Compare
|
/bot run --disable-fail-fast |
1 similar comment
|
/bot run --disable-fail-fast |
|
PR_Github #56512 [ run ] triggered by Bot. Commit: |
|
PR_Github #56512 [ run ] completed with state
|
dongfengy
changed the title
0b38f28 to
baf824c
Compare
dongfengy
changed the title
|
/bot run --disable-fail-fast |
|
PR_Github #56678 [ run ] triggered by Bot. Commit: |
|
PR_Github #56678 [ run ] completed with state
|
baf824c to
0dac1c9
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #56761 [ run ] triggered by Bot. Commit: |
|
PR_Github #56761 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #56850 [ run ] triggered by Bot. Commit: |
|
PR_Github #56850 [ run ] completed with state
|
0dac1c9 to
0cbb805
Compare
|
/bot run --disable-fail-fast |
|
PR_Github #57856 [ run ] triggered by Bot. Commit: |
|
PR_Github #57856 [ run ] completed with state
|
|
/bot run --disable-fail-fast |
|
PR_Github #57981 [ run ] triggered by Bot. Commit: |
|
PR_Github #57981 [ run ] completed with state |
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
0cbb805 to
306dd6e
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (1)
tests/unittest/llmapi/test_sampling_params.py (1)
216-371: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd Harmony conversion tests for
json_object,regex, andgrammar._guided_decoding_contentconverts all three types, but the tests only assert Harmony conversion forjson.Coverage summary: Added tests cover nine functions:
test_harmony_guided_decoding_triggers_on_final_channel,test_harmony_guided_decoding_accepts_json_schema_string,test_guided_decoding_preserves_top_level_schema_property,test_resolve_guided_decoding_reasoning_parser,test_resolve_raw_guided_decoding_reasoning_parser,test_raw_llm_preserves_guides_outside_harmony_xgrammar,test_plain_model_guided_decoding_is_unchanged,test_reasoning_parser_guided_decoding_uses_sequence_for_normal_parser, andtest_existing_structural_tag_guided_decoding_is_unchanged. The target file is listed intests/integration/test_lists/test-db/l0_cpu.yml. Coverage verdict: insufficient.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/llmapi/test_sampling_params.py` around lines 216 - 371, Add Harmony conversion tests for GuidedDecodingParams json_object, regex, and grammar inputs, exercising adapt_guided_decoding_params_for_reasoning_parser and asserting each produces the expected structural-tag content. Keep the existing JSON tests and surrounding parser-resolution coverage unchanged.Source: Path instructions
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tensorrt_llm/evaluate/json_mode_eval.py`:
- Around line 31-36: Update the imports in json_mode_eval.py to define the
annotation symbols used by JsonValue and the LLM-related code: import Union from
typing and import LLM from the existing LLM API module used by the file. Keep
the annotations and surrounding logic unchanged.
In `@tensorrt_llm/llmapi/reasoning_parser.py`:
- Around line 177-188: Define recursive JsonValue and structural-tag TypedDict
aliases, then update _normalize_json_schema_for_structural_tag to accept str |
BaseModel | dict[str, JsonValue] and return JsonValue. Change
_guided_decoding_content to return the structural-tag content TypedDict or None,
replacing Any and bare dict types without altering behavior.
---
Nitpick comments:
In `@tests/unittest/llmapi/test_sampling_params.py`:
- Around line 216-371: Add Harmony conversion tests for GuidedDecodingParams
json_object, regex, and grammar inputs, exercising
adapt_guided_decoding_params_for_reasoning_parser and asserting each produces
the expected structural-tag content. Keep the existing JSON tests and
surrounding parser-resolution coverage unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: be1fe29b-b3ed-4461-9a58-40eb955645af
📒 Files selected for processing (7)
tensorrt_llm/evaluate/interface.pytensorrt_llm/evaluate/json_mode_eval.pytensorrt_llm/llmapi/llm.pytensorrt_llm/llmapi/reasoning_parser.pytensorrt_llm/serve/openai_protocol.pytests/unittest/evaluate/test_generation_content.pytests/unittest/llmapi/test_sampling_params.py
🚧 Files skipped from review as they are similar to previous changes (4)
- tests/unittest/evaluate/test_generation_content.py
- tensorrt_llm/evaluate/interface.py
- tensorrt_llm/llmapi/llm.py
- tensorrt_llm/serve/openai_protocol.py
| def _normalize_json_schema_for_structural_tag(json_schema: Any) -> Any: | ||
| """Convert supported schema representations to structural-tag JSON.""" | ||
| if hasattr(json_schema, "model_json_schema"): | ||
| json_schema = json_schema.model_json_schema() | ||
| if isinstance(json_schema, str): | ||
| json_schema = json.loads(json_schema) | ||
| return json_schema | ||
|
|
||
|
|
||
| def _guided_decoding_content( | ||
| guided_decoding_params: GuidedDecodingParams) -> Optional[dict]: | ||
| """Translate an ordinary guide into structural-tag content.""" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
ast-grep outline tensorrt_llm/llmapi/reasoning_parser.py --items all --type function
rg -n -C 2 --glob '*.py' \
'GuidedDecodingParams\(|_normalize_json_schema_for_structural_tag' \
tensorrt_llm testsRepository: NVIDIA/TensorRT-LLM
Length of output: 21804
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- candidate files ---'
fd -i 'CODING_GUIDELINES.md|sampling_params.py|reasoning_parser.py' .
printf '%s\n' '--- GuidedDecodingParams declaration and imports ---'
rg -n -C 8 \
'class GuidedDecodingParams|json:|from pydantic|from typing|TypeAlias|JsonValue|JSON' \
tensorrt_llm/sampling_params.py tensorrt_llm/llmapi/reasoning_parser.py
printf '%s\n' '--- relevant implementation ---'
sed -n '1,35p;130,270p' tensorrt_llm/llmapi/reasoning_parser.py
sed -n '620,790p' tensorrt_llm/sampling_params.py
printf '%s\n' '--- relevant tests ---'
sed -n '190,285p;300,375p' tests/unittest/llmapi/test_sampling_params.py
printf '%s\n' '--- typing conventions ---'
rg -n -C 2 \
'JsonValue|JSONValue|TypeAlias|dict\[str, .*\]|BaseModel.*model_json_schema|model_json_schema' \
tensorrt_llm tests --glob '*.py' | head -n 240
printf '%s\n' '--- coding guidelines ---'
if [ -f CODING_GUIDELINES.md ]; then
sed -n '1,240p' CODING_GUIDELINES.md
else
printf '%s\n' 'CODING_GUIDELINES.md not found at repository root'
fiRepository: NVIDIA/TensorRT-LLM
Length of output: 50377
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 12 'class GuidedDecodingParams|json:' tensorrt_llm/sampling_params.py
sed -n '170,220p' tensorrt_llm/llmapi/reasoning_parser.py
sed -n '200,280p' tests/unittest/llmapi/test_sampling_params.py
rg -n -C 2 'JsonValue|JSONValue|TypeAlias|model_json_schema' tensorrt_llm tests --glob '*.py' | head -n 200Repository: NVIDIA/TensorRT-LLM
Length of output: 20848
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '%s\n' '--- declaration ---'
rg -n -C 15 'class GuidedDecodingParams|json:.*Optional|json:.*Union' tensorrt_llm/sampling_params.py
printf '%s\n' '--- helper implementation ---'
sed -n '165,225p' tensorrt_llm/llmapi/reasoning_parser.py
printf '%s\n' '--- tests ---'
sed -n '205,285p' tests/unittest/llmapi/test_sampling_params.py
printf '%s\n' '--- related aliases and schema typing ---'
rg -n -C 3 'JsonValue|JSONValue|TypeAlias|model_json_schema|BaseModel' tensorrt_llm tests --glob '*.py' | head -n 240Repository: NVIDIA/TensorRT-LLM
Length of output: 23122
🌐 Web query:
NVIDIA TensorRT-LLM sampling_params.py GuidedDecodingParams json Type BaseModel
💡 Result:
In NVIDIA TensorRT-LLM, the GuidedDecodingParams class is defined in tensorrt_llm/sampling_params.py to configure constraints for text generation [1][2]. The json field within this class accepts a value of type Union[str, BaseModel, dict] [1][3]. When providing a Pydantic BaseModel to the json parameter, TensorRT-LLM automatically handles the schema conversion, as the implementation internally checks for the BaseModel type and calls its model_json_schema method to extract the necessary schema for guided decoding [4]. This functionality allows developers to define complex generation constraints using familiar Pydantic models, which are then passed via SamplingParams to the model's generation methods [5][6]. Summary of GuidedDecodingParams.json type: Type: Optional[Union[str, BaseModel, dict]] [1][3] Purpose: Allows the user to specify a JSON schema (as a string, dictionary, or Pydantic model) to restrict the model's output format [1][5]. Usage: If a Pydantic BaseModel is provided, the library converts it to a JSON schema automatically before processing [4]. Note: Only one guide type (e.g., json, regex, grammar) can be effectively used in a single GuidedDecodingParams instance; the class includes a _validate method to enforce this constraint [1][2].
Citations:
- 1: https://github.com/NVIDIA/TensorRT-LLM/blob/main/tensorrt_llm/sampling_params.py
- 2: https://github.com/NVIDIA/TensorRT-LLM/blob/7a8bd87f/tensorrt_llm/sampling_params.py
- 3: https://github.com/NVIDIA/TensorRT-LLM/blob/a8c5955/tensorrt_llm/sampling_params.py
- 4: https://nvidia.github.io/TensorRT-LLM/0.20.0rc0/_modules/tensorrt_llm/sampling_params.html
- 5: https://nvidia.github.io/TensorRT-LLM/latest/features/guided-decoding.html
- 6: https://nvidia.github.io/TensorRT-LLM/latest/examples/llm_guided_decoding.html
Use precise types for the structural-tag helpers.
Define recursive JsonValue and structural-tag TypedDict aliases. Type schema inputs as str | BaseModel | dict[str, JsonValue], normalized schemas as JsonValue, and _guided_decoding_content as the structural-tag content type or None. Do not use Any or bare dict.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tensorrt_llm/llmapi/reasoning_parser.py` around lines 177 - 188, Define
recursive JsonValue and structural-tag TypedDict aliases, then update
_normalize_json_schema_for_structural_tag to accept str | BaseModel | dict[str,
JsonValue] and return JsonValue. Change _guided_decoding_content to return the
structural-tag content TypedDict or None, replacing Any and bare dict types
without altering behavior.
Sources: Coding guidelines, Learnings
Signed-off-by: Dongfeng Yu <dongfengy@nvidia.com>
|
/bot run --disable-fail-fast |
|
PR_Github #64838 [ run ] triggered by Bot. Commit: |
|
PR_Github #64838 [ run ] completed with state
|
2 participants
Dev Engineer Review
LLM.generate()path.QA Engineer Review
extract_final_content_from_generation, including raw output, explicitqwen3, non-Harmony parsers, token IDs, and explicit Harmony parsing.Description
Fix guided decoding for the raw Python
LLM.generate()path when GPT-OSS uses the Harmony protocol with xgrammar. Caller-provided JSON, regex, and grammar constraints are enriched with model-aware Harmony framing so they apply only after the final-channel marker instead of constraining the reasoning transcript.The serving path keeps its existing behavior while sharing the same reasoning-parser adapter. JSON-mode evaluation now scores parser-selected final content, covering plain output, explicit Harmony output, and explicit normal reasoning-parser output without guessing a parser for unrelated models.
Raw guided-decoding behavior remains unchanged for normal reasoning parsers and llguidance. This PR does not change the public API.
Test Coverage
059d1c9a76.059d1c9a76: passed.059d1c9a76: passed with JSON accuracy 78/100 and grammar accuracy 96/100 (threshold 50.369).PR Checklist
Please review the following before submitting your PR:
PR description clearly explains what and why. If using CodeRabbit's summary, please make sure it makes sense.
PR Follows TRT-LLM CODING GUIDELINES to the best of your knowledge.
Test cases are provided for new code paths (see test instructions)
If PR introduces API changes, an appropriate PR label is added - either
api-compatibleorapi-breaking. Forapi-breaking, includeBREAKINGin the PR title.Any new dependencies have been scanned for license and vulnerabilities
CODEOWNERS updated if ownership changes
Documentation updated as needed
Update tava architecture diagram if there is a significant design change in PR.
The reviewers assigned automatically/manually are appropriate for the PR.
Please check this after reviewing the above items as appropriate for this PR.
GitHub Bot Help
To see a list of available CI bot commands, please comment
/bot help.