[TRTLLM-14813][doc] Add Kimi K3 examples and deployment guide - #17333
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdded Kimi K3 deployment documentation, runtime configurations, quick-start and GSM8K jobs, accuracy sweeps, and serving performance benchmarks for Blackwell GPUs. Updated one DSA attention test fixture to initialize its draft-loop state. ChangesKimi K3 deployment support
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant Slurm
participant KimiK3Launcher
participant TensorRTLLM
participant trtllm_serve
participant BenchmarkClient
Slurm->>KimiK3Launcher: launch containerized multi-node inference
KimiK3Launcher->>TensorRTLLM: initialize model and generate samples
Slurm->>trtllm_serve: launch configured serving process
trtllm_serve-->>Slurm: report health status
BenchmarkClient->>trtllm_serve: send warmup and measured requests
BenchmarkClient-->>Slurm: save JSON benchmark results
Slurm->>trtllm_serve: terminate serving process
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 14
Note
Due to the large number of review comments, Critical, Major severity comments were prioritized as inline comments.
🟡 Minor comments (9)
docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md-1-1 (1)
1-1: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the required NVIDIA copyright headers.
The new guide has no header. The modified index has no header to update for the 2026 modification.
docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md#L1-L1: add the NVIDIA copyright and SPDX header.docs/source/deployment-guide/index.rst#L39-L39: add or update the NVIDIA copyright header with year2026.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md` at line 1, Add the required NVIDIA copyright and SPDX header to docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md at lines 1-1, and add or update the NVIDIA copyright header to year 2026 in docs/source/deployment-guide/index.rst at lines 39-39.Source: Coding guidelines
examples/kimi_k3/perf_sweep/submit_acc_sweep.sh-25-28 (1)
25-28: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate option values before reading
$2.With
set -u, an option without a value terminates the script with an unset-variable error instead of the documented usage error.
examples/kimi_k3/perf_sweep/submit_acc_sweep.sh#L25-L28: guard every value-taking option with[[ $# -ge 2 ]].examples/kimi_k3/perf_sweep/submit_perf_sweep.sh#L30-L33: guard every value-taking option with[[ $# -ge 2 ]].🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/kimi_k3/perf_sweep/submit_acc_sweep.sh` around lines 25 - 28, Guard every value-taking option in the argument-parsing logic of examples/kimi_k3/perf_sweep/submit_acc_sweep.sh at lines 25-28 and examples/kimi_k3/perf_sweep/submit_perf_sweep.sh at lines 30-33 with [[ $# -ge 2 ]] before reading $2, preserving the documented usage error for missing values under set -u.examples/kimi_k3/quick_start_kimi_k3.py-73-77 (1)
73-77: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winFail the quick-start job when a validation check fails.
The script only prints
Falsewhen generated text does not contain an expected value. It still exits with status0, so Slurm reports success for a failed functional check.Track failed checks and raise
SystemExitafter generation.Proposed fix
+ all_checks_passed = True try: for output, (_, expected) in zip(llm.generate(prompts, sampling_params), SAMPLES): generated_text = output.outputs[0].text + contains_expected_text = expected in generated_text print(f"Prompt: {output.prompt!r}") print(f"Generated text: {generated_text!r}") - print(f"Contains expected text {expected!r}: {expected in generated_text}\n") + print(f"Contains expected text {expected!r}: {contains_expected_text}\n") + all_checks_passed &= contains_expected_text + if not all_checks_passed: + raise SystemExit("One or more Kimi K3 quick-start checks failed") finally: llm.shutdown()🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/kimi_k3/quick_start_kimi_k3.py` around lines 73 - 77, Update the validation loop in the quick-start script to track whether any expected value is missing from generated_text, while preserving the existing output. After all generations complete, raise SystemExit with a nonzero status when a check failed so Slurm reports the job as unsuccessful.examples/kimi_k3/perf_sweep/acc_sweep.sbatch-29-42 (1)
29-42: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate option values before using them.
Both parsers read
$2without checking that an option has a value. Withset -u, a command such as--modelwithout a value exits with an unbound-variable error. The performance sweep also accepts invalid--isl,--osl, and concurrency values until a later shell or benchmark failure.
examples/kimi_k3/perf_sweep/acc_sweep.sbatch#L29-L42: check"$#"before reading$2.examples/kimi_k3/perf_sweep/perf_sweep.sbatch#L37-L54: check"$#"before reading$2, then validate positive integer input lengths and concurrencies.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/kimi_k3/perf_sweep/acc_sweep.sbatch` around lines 29 - 42, Update the argument parsers in examples/kimi_k3/perf_sweep/acc_sweep.sbatch (lines 29-42) and examples/kimi_k3/perf_sweep/perf_sweep.sbatch (lines 37-54) to verify an option value remains before reading $2, and emit the existing invalid-argument error for missing values. In perf_sweep.sbatch, additionally validate that ISL, OSL, and concurrency arguments are positive integers before they are used; apply no direct change beyond this validation to other sites.examples/kimi_k3/README.md-1-1 (1)
1-1: 📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick winAdd the required NVIDIA license headers to the new files.
examples/kimi_k3/README.md#L1-L1: add the repository-standard NVIDIA copyright and SPDX header.examples/kimi_k3/eval_extra_llm_options.yaml#L1-L1: add the repository-standard NVIDIA copyright and SPDX header.examples/kimi_k3/eval_extra_llm_options_reuse.yaml#L1-L1: add the repository-standard NVIDIA copyright and SPDX header.Based on learnings, new README files require a full NVIDIA copyright header.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/kimi_k3/README.md` at line 1, Add the repository-standard full NVIDIA copyright and SPDX license headers to examples/kimi_k3/README.md lines 1-1, examples/kimi_k3/eval_extra_llm_options.yaml lines 1-1, and examples/kimi_k3/eval_extra_llm_options_reuse.yaml lines 1-1, preserving each file’s existing content after the headers.Sources: Coding guidelines, Learnings
tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py-375-376 (1)
375-376: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winNarrow the expected exception type in the two mutation tests.
Both mutation tests use
pytest.raises(Exception, match="Mismatch percentage").Exceptionis the widest catchable type. If the mutated forward pass fails for an unrelated reason, for example a shape error inside the swapped-weight path, and the message happens to match, the test passes without proving the mutation changed the numerics.check_accuracysignals a tolerance failure withAssertionError. Expect that type.Based on coding guidelines: "Catch the narrowest exception possible... prefer built-in exception types".
🐛 Proposed fix
- with pytest.raises(Exception, match="Mismatch percentage"): + with pytest.raises(AssertionError, match="Mismatch percentage"): check_accuracy(out_fused, out_ref, atol=0.1, rtol=0.15, percent=0.95)Apply the same change at line 406. Confirm the type
check_accuracyraises intests/unittest/utils/util.pyfirst.Also applies to: 406-407
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py` around lines 375 - 376, Update both mutation tests around the `check_accuracy` calls, including the assertion at line 406, to expect `AssertionError` instead of the broad `Exception` type. Confirm `check_accuracy` in `tests/unittest/utils/util.py` raises `AssertionError` for tolerance failures, while preserving the existing `"Mismatch percentage"` match.Source: Coding guidelines
tests/unittest/_torch/modules/kimi_kda/test_kda_cache_soundness.py-33-49 (1)
33-49: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winAdd an import guard for the CuTe DSL dependencies.
The module guards only
flaand the Blackwell architecture.test_recycled_id_input_wrap_cacheimportscutlassat line 346, and_op_module()importscute_dsl_kimi_k3_custom_ops, which needsnvidia-cutlass-dslandcuda-bindings. On a Blackwell node without those packages, the tests error at collection or call time instead of skipping. The sibling filestests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.pyandtests/unittest/_torch/modeling/test_kimi_kda_fused_verify_parity.pyguard these dependencies explicitly.🛡️ Proposed fix
pytest.importorskip("fla") +pytest.importorskip("cutlass") +pytest.importorskip("cuda.bindings.driver")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/kimi_kda/test_kda_cache_soundness.py` around lines 33 - 49, Add an explicit import guard near the existing fla guard for the CuTe DSL dependencies required by _op_module() and test_recycled_id_input_wrap_cache, including nvidia-cutlass-dsl and cuda-bindings. Reuse the established guard pattern from the sibling parity tests so environments missing these packages skip the module during collection rather than failing at import or execution time.tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py-239-245 (1)
239-245: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick winInitialize
dt_biasinstead of leaving it uninitialized.
torch.empty(projection_size, dtype=torch.float32)leavesdt_biaswith arbitrary memory, which can include NaN or Inf. Every other parameter in this module is initialized (nn.Lineardefaults,A_logviauniform_,o_normviareset_parameters). If a run does not load a checkpoint, for example a dummy-weight or profiling run, the KDA gate silently produces NaN. Usetorch.zerosso an unloaded parameter yields a defined value.🐛 Proposed fix
- self.dt_bias = nn.Parameter(torch.empty(projection_size, dtype=torch.float32)) + self.dt_bias = nn.Parameter(torch.zeros(projection_size, dtype=torch.float32))🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py` around lines 239 - 245, Initialize the dt_bias parameter in the KimiKda mixer with zeros instead of torch.empty, preserving its float32 dtype and projection_size shape so unloaded or freshly constructed modules produce defined gate values.tensorrt_llm/models/quant_config_utils.py-63-71 (1)
63-71: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winValidate the weight fields on the
mxfp4-pack-quantizedformat path.The first disjunct trusts the
formatstring alone. The second disjunct validatesnum_bits,type, andstrategy. Because of the short circuit, a config that declares the format but carries different weight fields still selectsW4A16_MXFP4, and Line 69 then readsweights_quant_config["group_size"]. If that key is absent the caller sees a bareKeyErrorinstead of the descriptive errors this function raises everywhere else.Require the weight fields on both paths, or read
group_sizewith a checked lookup.🛡️ Proposed fix
if hf_quant_config.get("format") == "mxfp4-pack-quantized" or ( weights_quant_config["num_bits"] == 4 and weights_quant_config.get("type") == "float" and weights_quant_strategy == "group" and group_config.get("input_activations") is None ): - group_size = weights_quant_config["group_size"] + if weights_quant_config["num_bits"] != 4 or weights_quant_config.get("type") != "float": + raise ValueError( + "mxfp4-pack-quantized requires 4-bit float weights, got " + f"num_bits={weights_quant_config['num_bits']}, " + f"type={weights_quant_config.get('type')}." + ) + group_size = weights_quant_config.get("group_size") if group_size != 32: raise ValueError(f"Unsupported group_size: {group_size}. Supported: 32 for MXFP4.")🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/models/quant_config_utils.py` around lines 63 - 71, Update the MXFP4 selection condition around weights_quant_config and hf_quant_config so the mxfp4-pack-quantized format path also validates the required weight fields before selecting the configuration. Read group_size through a checked lookup or explicitly validate its presence, preserving the existing descriptive ValueError behavior instead of allowing a bare KeyError.
🧹 Nitpick comments (21)
tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py (1)
60-61: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winCorrect the optional field annotations.
num_shared_expertsandrouted_expert_hidden_sizeare annotatedintbut default toNone. The annotation contradicts the default and the usage:test_fused_matches_reference_with_latent_and_shared_expertsrelies on theNonedefault for the base case and overrides both fields at lines 330-332.Based on coding guidelines: "Annotate every function, use
Nonefor procedures, avoid unnecessaryAny... prefer built-in generic types and|".♻️ Proposed fix
- num_shared_experts: int = None - routed_expert_hidden_size: int = None + num_shared_experts: int | None = None + routed_expert_hidden_size: int | None = None🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.py` around lines 60 - 61, Update the annotations for num_shared_experts and routed_expert_hidden_size to use the optional integer form, preserving their None defaults and integer override behavior in test_fused_matches_reference_with_latent_and_shared_experts.Source: Coding guidelines
tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_state_parity.py (1)
293-315: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRename the dimension variable
kin_run_headcount_case.Line 295 binds
ktoHEAD_K_DIMwhile the key tensor is namedkey. Everywhere else in this fileknames the key tensor, for example in_runat line 85 and in_make_inputsat line 82. The two meanings collide within one module and make line 326 (scale=k**-0.5) hard to read. Rename the dimension tohead_k.Based on coding guidelines: "Avoid shadowing outer-scope variables" and "Use the same term for the same concept."
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_state_parity.py` around lines 293 - 315, Rename the head-dimension variable bound in _run_headcount_case from k to head_k, and update all dimension and scaling references in that function accordingly, including the scale=k**-0.5 expression. Keep k reserved for the key tensor to match the rest of the module.Source: Coding guidelines
tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_op.py (1)
28-46: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoffShare the attention pair through a module-scoped fixture.
_make_attention_pairbuilds twoKimiKDALinearAttentionmodules athidden_size=7168with 96 heads. The q/k/v/o projections alone account for well over one gigabyte of bf16 parameters per pair. Four tests call this helper independently, so the suite allocates and frees that memory four times. The sibling filestests/unittest/_torch/modules/kimi_kda/test_kda_prefill_state_parity.pyandtests/unittest/_torch/modules/kimi_kda/test_kda_cache_soundness.pyalready use ascope="module"dispatch_pairfixture for the same purpose. Convert this helper to a module-scoped fixture to cut allocation churn and runtime.♻️ Proposed fix
-def _make_attention_pair() -> tuple[KimiKDALinearAttention, KimiKDALinearAttention]: +@pytest.fixture(scope="module") +def attention_pair() -> tuple[KimiKDALinearAttention, KimiKDALinearAttention]: common = {Then take
attention_pairas a parameter in each test instead of calling the helper.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modules/kimi_kda/test_kda_prefill_op.py` around lines 28 - 46, Convert _make_attention_pair into a module-scoped pytest fixture, following the existing dispatch_pair pattern in the sibling Kimi KDA tests while preserving its module construction and assertions. Update every test that currently calls _make_attention_pair to accept the fixture as an attention_pair parameter and reuse the shared pair.tests/unittest/_torch/modeling/test_kimi_kda_fused_verify_parity.py (1)
164-169: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueMake the failure message identify the failing check.
_repreturns a bool and the test aggregates all results intook. A failure produces onlyassert ok. The reader must scan captured stdout to find which of the three comparisons failed. Raise inside_repwith the metric values instead.♻️ Proposed fix
def _rep(name, a, b): a, b = a.float(), b.float() cos = torch.nn.functional.cosine_similarity(a.flatten(), b.flatten(), dim=0).item() rel = ((a - b).norm() / (b.norm() + 1e-12)).item() - print(f" {name}: cos={cos:.6f} rel_l2={rel:.3e}") - return cos > 0.999 and rel < 3e-2 + assert cos > 0.999 and rel < 3e-2, f"{name}: cos={cos:.6f} rel_l2={rel:.3e}"Then replace each
ok &= _rep(...)with a plain_rep(...)call and dropok/assert ok.Also applies to: 229-229
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modeling/test_kimi_kda_fused_verify_parity.py` around lines 164 - 169, Update _rep to raise an assertion or equivalent failure directly when its cosine or relative L2 thresholds are not met, including the name and computed metric values in the failure message. Replace each ok &= _rep(...) call with a direct _rep(...) call, then remove the ok accumulator and final assert ok.tests/unittest/models/test_quant_config_utils.py (1)
289-313: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winAdd coverage for the inferred MXFP4 branch.
update_quant_config_from_compressed_tensorsselects MXFP4 through two independent conditions:format == "mxfp4-pack-quantized", ornum_bits == 4withtype == "float",strategy == "group", andinput_activations is None. This test setsformat, so only the first condition is exercised. The helper_compressed_tensors_configat line 37 substitutes a defaultinput_activationsdict whenever the argument is falsy, so no existing test can reach the second condition. A checkpoint that omitsformatwould take the untested path.♻️ Proposed addition
Change the helper so an explicit
Noneis preserved, then add the case:def _compressed_tensors_config( weights=None, input_activations=_UNSET, **overrides ): ... "input_activations": ( {"strategy": "tensor_group"} if input_activations is _UNSET else input_activations ),def test_update_quant_config_from_compressed_tensors_mxfp4_inferred_without_format(): quant_config = QuantConfig() update_quant_config_from_compressed_tensors( quant_config, _compressed_tensors_config( weights={ "num_bits": 4, "type": "float", "strategy": "group", "group_size": 32, }, input_activations=None, ignore=["lm_head"], ), ) assert quant_config.quant_algo == QuantAlgo.W4A16_MXFP4 assert quant_config.group_size == 32🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/models/test_quant_config_utils.py` around lines 289 - 313, Update _compressed_tensors_config to distinguish an omitted input_activations argument from an explicit None, preserving None instead of replacing it with the default dictionary. Add coverage for update_quant_config_from_compressed_tensors with MXFP4 weights, input_activations=None, and no format, asserting the inferred W4A16_MXFP4 algorithm and group size.tests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.py (1)
331-352: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueRead
Mfromdatainstead of the module global.
_fla_sequential_referenceunpacksB, H, K, V, WandTfromdata, but line 352 reads the module-levelM.make_conv_dataacceptsMas a parameter, so a future call withM != 2would silently desynchronize this reference fromcpu_reference, which usesdata["M"].♻️ Proposed fix
- B, H, K, V, W = data["B"], data["H"], data["K"], data["V"], data["W"] - T = data["T"] + B, H, K, V, W = data["B"], data["H"], data["K"], data["V"], data["W"] + T, M = data["T"], data["M"]🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.py` around lines 331 - 352, Update _fla_sequential_reference to read M from data, alongside the other unpacked dimensions, and use that local value in the range controlling the decode loop. Do not rely on the module-level M so calls with make_conv_data parameters other than 2 remain synchronized with cpu_reference.tensorrt_llm/_torch/pyexecutor/config_utils.py (1)
27-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd type annotations to the new public helpers.
is_kimi_linear,unwrap_kimi_text_config,get_kimi_linear_layer_masks, andget_kimi_linear_num_attention_layershave no parameter or return annotations. Add them, for exampledef is_kimi_linear(config) -> bool:anddef get_kimi_linear_layer_masks(config) -> tuple[list[bool], list[bool]]:.As per coding guidelines: "Annotate every function, use
Nonefor procedures".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/config_utils.py` around lines 27 - 84, Add parameter and return type annotations to the public helpers is_kimi_linear, unwrap_kimi_text_config, get_kimi_linear_layer_masks, and get_kimi_linear_num_attention_layers. Use bool for the predicate, tuple[list[bool], list[bool]] for the layer-mask result, and appropriate config/object and integer return types for the remaining helpers, following the guideline to annotate every function.Source: Coding guidelines
tensorrt_llm/_torch/modules/fused_moe/moe_op_backend.py (1)
337-367: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConsider moving the SiTu validation off the per-call path.
run_fp4_block_scale_moeruns once per MoE layer per forward step. These checks are static properties of the module configuration and the loaded weights, andTRTLLMGenFusedMoE._check_configsalready validates most of them once at weight creation. Keeping them here adds Python work inside the decode loop. Gate them behind a debug flag, or validate once at setup and keep only the argument-level checks the op backend cannot see elsewhere.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/fused_moe/moe_op_backend.py` around lines 337 - 367, The SiTu validation block in run_fp4_block_scale_moe performs static module and weight checks on every forward call. Move these checks into TRTLLMGenFusedMoE._check_configs or another one-time setup path, leaving only per-call argument validation that the backend cannot establish during configuration; alternatively guard the static checks behind the existing debug-validation mechanism.tensorrt_llm/_torch/models/checkpoints/hf/weight_loader.py (1)
244-270: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueCorrect the return annotation to
ConsumableWeightsDict.The function returns
ConsumableWeightsDict(weights), but the signature declaresdict[str, Any]and the docstring says "a dict". Model loaders callmark_consumed()on the returned object, so the precise type matters for callers and type checkers. The sibling loaders (_prefetch_and_load,_load_weights_in_parallel) already annotateConsumableWeightsDict.♻️ Proposed annotation fix
- def _load_lazy_safetensors(self, checkpoint_dir: str) -> dict[str, Any]: - """Return a dict of name -> lazy safetensors slices. + def _load_lazy_safetensors(self, + checkpoint_dir: str) -> ConsumableWeightsDict: + """Return a consumable mapping of name -> lazy safetensors slices.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/models/checkpoints/hf/weight_loader.py` around lines 244 - 270, Update the return annotation of _load_lazy_safetensors to ConsumableWeightsDict and revise its docstring return description to reflect that concrete type, while preserving the existing ConsumableWeightsDict(weights) return behavior.Source: Coding guidelines
tensorrt_llm/_torch/models/modeling_kimi_linear.py (1)
2573-2575: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReuse the
kda_fp8flag instead of re-reading the environment variable.Line 2574 computes
kda_fp8from_KIMI_K3_FP8_WEIGHT_READ_KDA_ENV, and line 2612 evaluates the same condition again. The two conditions must stay identical: line 2583 skipsfinalize_decode_weights()whenkda_fp8is true, and line 2612 decides whether the FP8 KDA modules are actually built. If the two expressions ever diverge, the KDA layers keep neither the bf16 fused decode path nor the FP8 projections. Use the already-computed flag.♻️ Proposed change
- if os.environ.get(_KIMI_K3_FP8_WEIGHT_READ_KDA_ENV, "1") != "0": + if kda_fp8: n_kda = _convert_kda_projections_to_fp8_weight_read(self.model)Also applies to: 2612-2618
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/models/modeling_kimi_linear.py` around lines 2573 - 2575, Update the FP8 KDA module-building condition near the existing `kda_fp8` logic to reuse the already-computed `kda_fp8` flag instead of re-reading `_KIMI_K3_FP8_WEIGHT_READ_KDA_ENV`. Keep the surrounding `kda_glue_fp8` behavior and the `finalize_decode_weights()` decision unchanged.tensorrt_llm/_torch/configs/kimi_linear.py (1)
107-107: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winReplace the config asserts with explicit
ValueErrorraises.
python -Ostripsassert, so these checks disappear in optimized runs. A malformedlinear_attn_configthen reachesis_kda_layer()and raises a bareKeyError. Explicit raises also give an actionable message.♻️ Proposed validation change
- self.moe_router_activation_func = moe_router_activation_func - assert self.moe_router_activation_func in ("softmax", "sigmoid") + self.moe_router_activation_func = moe_router_activation_func + if self.moe_router_activation_func not in ("softmax", "sigmoid"): + raise ValueError( + "moe_router_activation_func must be 'softmax' or 'sigmoid', " + f"got {self.moe_router_activation_func!r}" + )if linear_attn_config is not None: - assert linear_attn_config["kda_layers"] is not None - assert linear_attn_config["full_attn_layers"] is not None + for key in ("kda_layers", "full_attn_layers"): + if linear_attn_config.get(key) is None: + raise ValueError(f"linear_attn_config is missing '{key}'")Also applies to: 124-127
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/configs/kimi_linear.py` at line 107, Replace the configuration asserts in the Kimi linear config validation, including the checks near moe_router_activation_func and lines 124-127, with explicit ValueError raises that remain active under python -O. Preserve the existing validation conditions and include actionable messages identifying the invalid configuration field and accepted values, preventing malformed linear_attn_config from reaching is_kda_layer().Source: Coding guidelines
tensorrt_llm/_torch/modules/kimi_k3_moe/_mlp.py (1)
111-149: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winValidate the op inputs with explicit exceptions, not
assert.
situ_and_mulis a registered public op (torch.ops.trtllm.situ_and_mul). Two contract requirements are unchecked at runtime:
assert n % 2 == 0disappears underpython -O. An odd last dim then truncatesdand the kernel reads the wrongupslice, producing silently wrong activations.- The kernel indexes
x_row_ptr + offsetsandx_row_ptr + offsets + d, which assumesx.stride(-1) == 1. The docstring states this, but nothing enforces it. A non-contiguous input produces wrong results instead of an error.🛡️ Proposed validation
b, n = x.shape - assert n % 2 == 0 + if n % 2 != 0: + raise ValueError(f"situ_and_mul expects an even last dim, got {n}") + if x.stride(-1) != 1: + raise ValueError("situ_and_mul expects the last dim of x to be contiguous") d = n // 2🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_k3_moe/_mlp.py` around lines 111 - 149, Update situ_and_mul to validate its public-op input contract with explicit exceptions: reject tensors whose last dimension is odd, and reject tensors whose final-dimension stride is not 1, before computing d or launching situ_and_mul_kernel. Preserve the existing behavior for valid contiguous-last-dimension inputs and use clear error messages describing each violated requirement.Source: Coding guidelines
tensorrt_llm/mapping.py (1)
104-114: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winPreserve MoE TP/EP provenance across mapping serialization.
Mapping.to_dict()emits resolved values, soMapping.from_dict()marks an auto mapping as explicit.ModelConfiguses this path.KimiK3MoERuntime._select_moe_tp_ep()then changes the default split from(1, tp_size)to(tp_size, 1), changing expert checkpoint sharding. Serialize the provenance flag or preserve the-1sentinels, and add a round-trip regression test.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/mapping.py` around lines 104 - 114, Preserve the original auto-versus-explicit MoE TP/EP provenance through Mapping.to_dict()/from_dict() instead of recomputing it from resolved sizes. Serialize and restore moe_tp_ep_user_specified (or retain the -1 sentinels), ensure ModelConfig round-trips keep auto mappings on the default (1, tp_size) split, and add a regression test covering this round trip and KimiK3MoERuntime._select_moe_tp_ep().tensorrt_llm/_torch/modules/kimi_k3_attn_res/__init__.py (1)
3-16: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUpdate the module docstring to describe the in-tree op.
The docstring points at
exisiting_optimization_work/Attention_residualas the kernel source._attn_res_kernels.pystates the kernel is source-integrated astrtllm::attn_res_fwdundercpp/tensorrt_llm/kernels/kimiK3AttnRes. Align this docstring with the in-tree path so readers do not look for an external tree.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_k3_attn_res/__init__.py` around lines 3 - 16, Update the module docstring to identify the source-integrated `trtllm::attn_res_fwd` kernel under `cpp/tensorrt_llm/kernels/kimiK3AttnRes` instead of referencing `exisiting_optimization_work/Attention_residual`; leave the remaining behavior and fallback description unchanged.tensorrt_llm/_torch/modules/kimi_kda/_kda_decode.py (1)
21-55: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBound the dummy-tensor cache.
_DUMMY_CACHEis a module-level dict that never evicts. Theonorm_gkey includes the batch sizeB(Line 147), so each distinct batch size adds a CUDA allocation that lives for the process lifetime. WithHV=96and large batches each entry is several MB. Add an eviction policy, or keyonorm_gon a single max-size buffer and slice it.Also add a precise type for the module-level annotation instead of bare
dict.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_kda/_kda_decode.py` around lines 21 - 55, Bound the module-level _DUMMY_CACHE used by _dummy_tensor so varying onorm_g batch sizes cannot retain unbounded CUDA allocations; either evict entries with a clear bounded policy or reuse one maximum-size buffer and slice it. Replace the bare dict annotation with a precise key/value type matching the cache contents.tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py (1)
95-101: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBroad
except Exceptionin both in-tree availability probes. Both probes swallow every exception, so a real failure inside an imported CuTe DSL module is reported as "kernel not available" and the run silently falls back to FLA. Ruff reports BLE001 at both sites.
tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py#L95-L101: catchImportErrorinis_intree_prefill_available.tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py#L134-L140: catchImportErrorinis_intree_mtp_available.As per coding guidelines: "Catch specific exceptions instead of using broad or bare
except:handlers."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py` around lines 95 - 101, Replace the broad exception handlers in is_intree_prefill_available and is_intree_mtp_available with ImportError handlers so only unavailable-module imports trigger the fallback; allow other initialization failures to propagate. Apply this change at tensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.py lines 95-101 and 134-140.Sources: Coding guidelines, Linters/SAST tools
tensorrt_llm/_torch/modules/kimi_k3_attn_res/_attn_res_kernels.py (1)
70-84: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueAdd the return type annotation.
intree_attn_res_fwdhas no return annotation. The coding guidelines require annotating every function. The op returns four tensors.♻️ Proposed annotation
def intree_attn_res_fwd( layer_residual: torch.Tensor, block_residual: torch.Tensor, res_weight: torch.Tensor, rms_weight: torch.Tensor, rms_eps: float, -): +) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]:As per coding guidelines: "Annotate every function, use
Nonefor procedures".🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_k3_attn_res/_attn_res_kernels.py` around lines 70 - 84, Add a return type annotation to intree_attn_res_fwd indicating that it returns a four-element tuple of torch.Tensor values, matching the output of torch.ops.trtllm.attn_res_fwd.Source: Coding guidelines
tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py (2)
489-497: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winCache the transposed conv weights instead of rebuilding them per decode step.
w_q_t_full,w_k_t_full, andw_v_t_fullrunsqueeze,transpose,to(bfloat16), andcontiguous()on everyforward_decodecall. Eachcontiguous()allocates and copies a constant tensor, once per layer per token._forward_decodeintensorrt_llm/_torch/models/modeling_kimi_linear.py(Lines 1289-1440) caches these asself._w_q_t,self._w_k_t, andself._w_v_tfor exactly this reason. Cache them here after the first call, or after weight loading. The same applies toA_log_full,dt_bias_full, andonorm_weight_fullat Lines 544-546.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py` around lines 489 - 497, Cache the transformed convolution weights and state tensors used by forward_decode instead of recomputing them each call. Update the relevant KimiKda mixer initialization or first-use path to populate and reuse self._w_q_t, self._w_k_t, self._w_v_t, and equivalent cached values for A_log_full, dt_bias_full, and onorm_weight_full, while preserving the existing bfloat16, transpose, and contiguous transformations.
51-69: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDuplicated
_meta_safe_cast_dtypehelper in two modules. Both Kimi K3 modules define a byte-identical meta-safe dtype cast helper, including the redundant localimport torch as _torchand the missing type annotations. The shared root cause is one helper copied instead of shared.
tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py#L51-L69: remove the local copy and import the shared helper.tensorrt_llm/_torch/modules/kimi_k3_mla/kimi_k3_mla_attention.py#L26-L44: remove the local copy and import the shared helper.Place the single definition in a shared module, drop the redundant
import torch as _torch, and annotate themoduleanddtypeparameters.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py` around lines 51 - 69, Remove the duplicated _meta_safe_cast_dtype definitions from tensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.py lines 51-69 and tensorrt_llm/_torch/modules/kimi_k3_mla/kimi_k3_mla_attention.py lines 26-44, and import the single shared helper from a common module in both files. Define it once with module and dtype type annotations, preserving the existing meta-safe casting behavior and eliminating the local import torch as _torch.tensorrt_llm/_torch/pyexecutor/_util.py (1)
158-182: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueDocument that this branch bypasses the manager-preference overrides.
The Kimi K3 branch returns before the
TRTLLM_USE_PY_MAMBAandTLLM_MAMBA_MANAGER_PREFERENCEhandling below. A user who setsTLLM_MAMBA_MANAGER_PREFERENCE=CPPwith block reuse disabled still getsMixedMambaHybridCacheManager, with no warning. State this in the comment, or move the Kimi branch after the override handling.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/pyexecutor/_util.py` around lines 158 - 182, Update the Kimi K3 branch comment near is_kimi_linear to explicitly state that its early returns bypass the TRTLLM_USE_PY_MAMBA and TLLM_MAMBA_MANAGER_PREFERENCE overrides, including that CPP preference is ignored when block reuse is disabled; alternatively, move this branch after the manager-preference handling while preserving its routing behavior.tensorrt_llm/_torch/modules/kimi_k3_moe/_moe_kernels.py (1)
374-407: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winUse named arguments and
RoutingMethodType.Renormalize. The custom op defines named parameters and supports keyword arguments. Pass the scalar parameters by keyword to prevent positional shifts. Replace1withint(RoutingMethodType.Renormalize). Thealignmentkeyword is supported bymxfp8_quantize.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tensorrt_llm/_torch/modules/kimi_k3_moe/_moe_kernels.py` around lines 374 - 407, Update the mxe4m3_mxe2m1_block_scale_moe_runner call to pass its scalar parameters using their defined keyword names, preventing positional argument shifts while preserving the existing values. Replace the literal routing method value 1 with int(RoutingMethodType.Renormalize), and retain the alignment keyword on mxfp8_quantize.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 2fb6dafb-454a-41d5-9867-abea81078d92
📒 Files selected for processing (63)
cpp/tensorrt_llm/batch_manager/kv_cache_manager_v2/blockRadixTree.cppcpp/tensorrt_llm/thop/kdaDecodeOp.cppdocs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.mddocs/source/deployment-guide/index.rstexamples/kimi_k3/README.mdexamples/kimi_k3/eval_extra_llm_options.yamlexamples/kimi_k3/eval_extra_llm_options_reuse.yamlexamples/kimi_k3/perf_sweep/acc_sweep.sbatchexamples/kimi_k3/perf_sweep/perf_sweep.sbatchexamples/kimi_k3/perf_sweep/submit_acc_sweep.shexamples/kimi_k3/perf_sweep/submit_perf_sweep.shexamples/kimi_k3/quick_start_kimi_k3.pyexamples/kimi_k3/quick_start_kimi_k3.sbatchexamples/kimi_k3/run_gsm8k_kimi_k3.sbatchexamples/kimi_k3/run_serving_benchmark_kimi_k3.sbatchtensorrt_llm/_torch/configs/__init__.pytensorrt_llm/_torch/configs/kimi_linear.pytensorrt_llm/_torch/custom_ops/cpp_custom_ops.pytensorrt_llm/_torch/model_config.pytensorrt_llm/_torch/models/__init__.pytensorrt_llm/_torch/models/checkpoints/hf/weight_loader.pytensorrt_llm/_torch/models/modeling_kimi_linear.pytensorrt_llm/_torch/modules/fused_moe/communication/communication_factory.pytensorrt_llm/_torch/modules/fused_moe/communication/deep_ep_low_latency.pytensorrt_llm/_torch/modules/fused_moe/configurable_moe.pytensorrt_llm/_torch/modules/fused_moe/create_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_trtllm_gen.pytensorrt_llm/_torch/modules/fused_moe/moe_op_backend.pytensorrt_llm/_torch/modules/fused_moe/moe_scheduler.pytensorrt_llm/_torch/modules/fused_moe/quantization.pytensorrt_llm/_torch/modules/kimi_k3_attn_res/__init__.pytensorrt_llm/_torch/modules/kimi_k3_attn_res/_attn_res_kernels.pytensorrt_llm/_torch/modules/kimi_k3_attn_res/kimi_k3_attn_res.pytensorrt_llm/_torch/modules/kimi_k3_mla/__init__.pytensorrt_llm/_torch/modules/kimi_k3_mla/kimi_k3_mla_attention.pytensorrt_llm/_torch/modules/kimi_k3_moe/__init__.pytensorrt_llm/_torch/modules/kimi_k3_moe/_mlp.pytensorrt_llm/_torch/modules/kimi_k3_moe/_moe_kernels.pytensorrt_llm/_torch/modules/kimi_k3_moe/_mxfp4.pytensorrt_llm/_torch/modules/kimi_k3_moe/kimi_k3_moe_block.pytensorrt_llm/_torch/modules/kimi_k3_moe/kimi_k3_moe_gate.pytensorrt_llm/_torch/modules/kimi_kda/__init__.pytensorrt_llm/_torch/modules/kimi_kda/_kda_decode.pytensorrt_llm/_torch/modules/kimi_kda/_kda_kernels.pytensorrt_llm/_torch/modules/kimi_kda/kimi_kda_mixer.pytensorrt_llm/_torch/modules/mamba/mamba2_metadata.pytensorrt_llm/_torch/modules/mla.pytensorrt_llm/_torch/pyexecutor/_util.pytensorrt_llm/_torch/pyexecutor/config_utils.pytensorrt_llm/_torch/pyexecutor/mamba_cache_manager.pytensorrt_llm/_torch/pyexecutor/resource_manager.pytensorrt_llm/_torch/utils.pytensorrt_llm/mapping.pytensorrt_llm/models/quant_config_utils.pytests/unittest/_torch/modeling/test_kda_mtp_decode_cute_parity.pytests/unittest/_torch/modeling/test_kimi_kda_fused_verify_parity.pytests/unittest/_torch/modeling/test_kimi_kda_verify_parity.pytests/unittest/_torch/modules/kimi_k3_attn_res/test_attn_res_op.pytests/unittest/_torch/modules/kimi_kda/test_kda_cache_soundness.pytests/unittest/_torch/modules/kimi_kda/test_kda_prefill_op.pytests/unittest/_torch/modules/kimi_kda/test_kda_prefill_state_parity.pytests/unittest/_torch/modules/moe/test_kimi_k3_situ_moe.pytests/unittest/models/test_quant_config_utils.py
Add the Kimi K3 example suite and the Blackwell deployment guide: - docs/source/deployment-guide/deployment-guide-for-kimi-k3-on-trtllm.md (linked from the deployment-guide index): build-from-source steps, DEP16/TEP16/TEP8 serving recipes, launch/eval/benchmark walkthroughs. - examples/kimi_k3: LLM API quick start (script + Slurm job), GSM8K evaluation job, single-recipe serving benchmark, and the perf_sweep/ performance and accuracy sweep drivers with tuned per-mode configs. Scripts take the checkpoint and container image as arguments and use placeholder Slurm partition/account settings; a reader fills in cluster-specific values at submit time. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
… drift and README reuse-default wording Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/kimi_k3/eval_extra_llm_options_reuse.yaml`:
- Around line 1-4: Add the repository-standard NVIDIA copyright header at the
beginning of eval_extra_llm_options_reuse.yaml, before the existing
configuration comments, using the latest meaningful modification year. Preserve
all existing YAML content and comments unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: c835191c-d200-43c6-b346-72193fc5ff06
📒 Files selected for processing (3)
examples/kimi_k3/README.mdexamples/kimi_k3/eval_extra_llm_options_reuse.yamlexamples/kimi_k3/run_gsm8k_kimi_k3.sbatch
🚧 Files skipped from review as they are similar to previous changes (1)
- examples/kimi_k3/README.md
test_on_update_kv_lens_rebuilds_stale_map builds its metadata with object.__new__, bypassing __init__ where in_mtp_draft_loop is initialized. Since NVIDIA#16925 made on_update_kv_lens() read that flag, the test fails with AttributeError on every pre-merge run. Set the __init__ default on the stub. Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
|
/bot run |
|
PR_Github #64586 [ run ] triggered by Bot. Commit: |
|
PR_Github #64586 [ run ] completed with state
|
|
/bot run |
Validation summary (out-of-CI, Blackwell nodes)
|
|
PR_Github #64610 [ run ] triggered by Bot. Commit: |
|
PR_Github #64610 [ run ] completed with state
|
|
/bot run |
|
PR_Github #64646 [ run ] triggered by Bot. Commit: |
|
PR_Github #64646 [ run ] completed with state
|
|
/bot run --disable-fail-fast --stage-list "A100X-PyTorch-1,B300-PyTorch-1,DGX_B200-PyTorch-5,DGX_H100-PyTorch-3,DGX_H100-PyTorch-4,DGX_H100-PyTorch-6,H100_PCIe-AutoDeploy-1,H100_PCIe-PyTorch-Ray-1" |
|
PR_Github #64685 [ run ] triggered by Bot. Commit: |
|
PR_Github #64685 [ run ] completed with state |
|
/bot run |
|
PR_Github #64721 [ run ] triggered by Bot. Commit: |
…LM-14814 Signed-off-by: Brian Nguyen <brnguyen@nvidia.com>
|
/bot skip --comment "Doc-only delta on top of 5db3c65: adds a note that the chat-completions example depends on TRTLLM-14814. 5db3c65 CI: full run (pipeline 52469) failed only on test_llm_context_only_timed_out_kv_cache_exhausted[PYTHON-NIXL-1000], an intermittent failure unrelated to this doc PR (same signature seen elsewhere; sibling param waived under nvbugs/6490004); stage-list rerun of the affected stages on the same commit (pipeline 52541) passed." |
|
PR_Github #64738 [ skip ] triggered by Bot. Commit: |
|
PR_Github #64721 [ run ] completed with state |
|
PR_Github #64738 [ skip ] completed with state |
9 participants
Description
Adds the Kimi K3 user-facing material that accompanies the model PR:
GSM8K evaluation job, serving benchmark job, eval option yamls, and the
perf/accuracy sweep scripts
Base: #17269 (KimiLinear model) has merged; this PR is rebased onto
main and carries only the docs/examples changes.
Notes
material is excluded; it ships with the PRs that add those features. The
README and guide describe them as not yet supported.
of the base model PR ([TRTLLM-14813][feat] Add Kimi K3 (KimiLinear) model #17269). Restore when the fp8-kv support PR lands.
used for bring-up; they are indicative, not a benchmark commitment.
top of each script and are cluster-agnostic (
bash -n,py_compile, andyaml parsing all verified).
Test Coverage
Docs and examples only; no functional code changes. Scripts validated with
bash -n/python3 -m py_compile/ yaml parse; pre-commit green.PR Checklist
[TRTLLM-14813][doc]conventionSummary
Dev Engineer Review
EXTRA_LLM_API_FILEas read-only.sbatchfailures.in_mtp_draft_loopbeforeon_update_kv_lens()reads it.QA Engineer Review
tests/unittest/_torch/attention/sparse/dsa/test_req_idx_per_token.py.md.in_mtp_draft_loop = False.tests/integration/test_lists/based on the provided changes.