[TRTLLM-14813][doc] Add Kimi K3 examples and deployment guide

docs/source/deployment-guide/index.rst

@@ -36,5 +36,6 @@ The deployment guides below provide more detailed instructions for serving speci
    deployment-guide-for-qwen3-on-trtllm.md
    deployment-guide-for-qwen3.5-on-trtllm.md
    deployment-guide-for-kimi-k2-thinking-on-trtllm.md
+   deployment-guide-for-kimi-k3-on-trtllm.md
    deployment-guide-for-glm-5-on-trtllm.md
    deployment-guide-for-minimax-m3-on-trtllm.md

examples/kimi_k3/README.md

@@ -0,0 +1,196 @@
+# Kimi K3
+
+This example runs Kimi K3 with TensorRT-LLM. It includes an LLM API quick
+start and configuration for GSM8K evaluation.
+
+## Hardware support
+
+Only NVIDIA Blackwell GPUs are currently supported and tested. Support for
+other GPU architectures may be added in a future release.
+
+## Prerequisites
+
+- TensorRT-LLM built from this repository and installed. Inside
+  the TensorRT-LLM container, from the repository root:
+
+  ```bash
+  python3 scripts/build_wheel.py --cuda_architectures 103-real --skip_building_wheel --yes
+  .venv-3.12/bin/python -m pip install --no-deps -e .
+  ```
+  Using editable mode is recommended for development and testing; see
+  [build from source](../../docs/source/installation/build-from-source.md)
+  for details.
+
+  `build_wheel.py` creates the virtual environment at the repository
+  root, named after the container's Python version: `.venv-3.12` for the
+  current containers (Python 3.12). If your container ships a different
+  Python, substitute the matching `.venv-<major>.<minor>` path in the
+  commands on this page. Adjust `--cuda_architectures` to the target
+  GPUs (`103-real` for GB300).
+- A complete Hugging Face-format Kimi K3 checkpoint and tokenizer, e.g.
+  [moonshotai/Kimi-K3](https://huggingface.co/moonshotai/Kimi-K3) downloaded
+  from the Hugging Face Hub (the example scripts take a local filesystem
+  path).
+- A Slurm cluster with 16 NVIDIA Blackwell GPUs and a TensorRT-LLM container
+  image.
+- The `fla-core` and `einops` packages, installed into the same in-place
+  environment. Note: these dependencies might be removed in a future
+  release, replaced by other kernels.
+
+  ```bash
+  .venv-3.12/bin/python -m pip install fla-core einops
+  ```
+- To use the optimized CuTeDSL MLA kernel, install the following FlashInfer
+  revision into the same in-place environment after installing TensorRT-LLM:
+
+  ```bash
+  .venv-3.12/bin/python -u -m pip install --force-reinstall --no-deps \
+      --no-build-isolation \
+      "flashinfer-python[cu13] @ git+https://github.com/PerkzZheng/flashinfer-k3.git@b6cc594918baf76c40c3a6236fd53f0f8fb9d2dc"
+  ```
+
+  The `packaging>=24.2` requirement of this source build is already
+  satisfied by `requirements.txt`. The TensorRT-LLM environment already
+  provides FlashInfer's runtime
+  dependencies; `--no-deps` prevents pip from replacing its pinned PyTorch,
+  Triton, CUDA, and CuTeDSL packages. Install FlashInfer last: TensorRT-LLM
+  currently pins `flashinfer-python==0.6.16`, so a later
+  dependency-resolving TensorRT-LLM install can replace this source revision.
+
+## Run the model
+
+Kimi K3 requires a multi-node launch. From the repository root, submit the
+quick-start Slurm job with the checkpoint and container paths:
+
+```bash
+sbatch examples/kimi_k3/quick_start_kimi_k3.sbatch \
+    --model /path/to/kimi-k3-checkpoint \
+    --image /path/to/tensorrt-llm-container.sqsh
+```
+
+Slurm prints the submitted job ID immediately. After the job starts, its output
+is written to `kimi-k3-quick-start-<job-id>.log` in the submission directory.
+The model is loaded once, then the log shows four prompts, their generated
+text, and whether each response contains the expected text. A successful run
+should report `True` for all four checks.
+
+For a full GSM8K evaluation, submit:
+
+```bash
+sbatch examples/kimi_k3/run_gsm8k_kimi_k3.sbatch \
+    --model /path/to/kimi-k3-checkpoint \
+    --image /path/to/tensorrt-llm-container.sqsh
+```
+
+This job writes progress and results to
+`kimi-k3-gsm8k-<job-id>.log`. If no local dataset path is configured,
+`trtllm-eval` downloads GSM8K from the Hugging Face Hub. The completed log
+contains a results table with the normalized GSM8K exact-match scores. With
+the tested checkpoint and the settings in this example, users should expect
+approximately:
+
+| Filter | Exact match |
+| :-- | --: |
+| Flexible extract | 96.51 |
+| Strict match | 96.44 |
+
+The expected average accuracy is approximately 96.47. Small differences are
+possible with different checkpoint or dependency revisions.
+
+For serving performance, use the standard sweep under
+`examples/kimi_k3/perf_sweep/` (this supersedes the older
+`run_serving_benchmark_kimi_k3.sbatch` single-recipe benchmark). It submits
+the 17-point 8K/1K sweep across the three tuned serving recipes — `tep16`
+(latency, c 1–16), `tep8` (interactive, 8 GPUs, c 1–16), and `dep16`
+(throughput, c 16–1024):
+
+```bash
+examples/kimi_k3/perf_sweep/submit_perf_sweep.sh \
+    --model /path/to/kimi-k3-checkpoint \
+    --image /path/to/tensorrt-llm-container.sqsh
+```
+
+All jobs of a comparison batch are submitted together on purpose: a weight
+load overlapping another job's measurement window depresses DEP16 c>=128
+points by ~30%. Use `--jobs "tep16 tep8 dep16-lo dep16-hi"` to select a
+subset and `--dry-run` to inspect the sbatch commands.
+
+Guard the same three recipes with the GSM8K accuracy sweep whenever the
+serving configs or kernels change (expect ~96.5 +/- 0.5 per recipe):
+
+```bash
+examples/kimi_k3/perf_sweep/submit_acc_sweep.sh \
+    --model /path/to/kimi-k3-checkpoint \
+    --image /path/to/tensorrt-llm-container.sqsh
+```
+
+Results land in per-job `kimi-k3-sweep-<name>-<job-id>.log` files and result
+JSONs in the submission directory (run from a fresh results folder).
+
+Scheduler options must precede the script path; model and image arguments
+follow it. The scripts default to the `batch` partition and use an `${account}`
+placeholder. Change the corresponding `#SBATCH` settings
+to match your cluster, or override them when submitting the job. For example:
+
+```bash
+sbatch --partition=PARTITION --account=ACCOUNT --time=04:00:00 \
+    SCRIPT --model MODEL --image IMAGE
+```
+
+## Troubleshooting
+
+### The job reaches its time limit
+
+The default time limits are intentionally aggressive to make the jobs easier
+to schedule: 40 minutes for the quick start and two hours for GSM8K.
+Depending on how fast your cluster's filesystem loads the weights, a job may
+be terminated before producing its final results, particularly with cold
+runtime caches or a busy filesystem.
+
+Request a longer allocation if the time is not sufficient for your environment:
+
+```bash
+sbatch --time=02:00:00 examples/kimi_k3/quick_start_kimi_k3.sbatch \
+    --model MODEL --image IMAGE
+
+sbatch --time=04:00:00 examples/kimi_k3/run_gsm8k_kimi_k3.sbatch \
+    --model MODEL --image IMAGE
+```
+
+## Chunked prefill and KV-cache block reuse
+
+Chunked prefill is supported and enabled by default in this example
+(`enable_chunked_prefill: true` in the quick start and in
+`eval_extra_llm_options.yaml`).
+
+KV-cache block reuse is supported. The LLM API enables it by default
+(`KvCacheConfig.enable_block_reuse` defaults to `true`), but the example
+configurations here explicitly disable it and treat reuse as an opt-in:
+set `kv_cache_config.enable_block_reuse: true`, or use the example
+flags — `--enable-block-reuse` for the quick start and `--reuse` for the
+GSM8K job (which selects `eval_extra_llm_options_reuse.yaml`):
+
+```bash
+sbatch examples/kimi_k3/quick_start_kimi_k3.sbatch \
+    --model MODEL --image IMAGE --enable-block-reuse
+
+sbatch examples/kimi_k3/run_gsm8k_kimi_k3.sbatch \
+    --model MODEL --image IMAGE --reuse
+```
+
+Unless one of the flags above is passed, these examples run with block
+reuse disabled; the tested evaluation and serving configurations use the
+default cache manager.
+
+## Current limitations
+
+- Pipeline parallelism is not supported.
+- **Known performance limitation at DEP16 saturation** (attention-DP +
+  EP16 throughput recipe): the 8K/1K serving sweep loses several percent
+  of output throughput at concurrency ≥ 128 (up to ~15% at concurrency
+  1024) relative to earlier development measurements; concurrency ≤ 64
+  and the TEP16/TEP8 latency recipes are unaffected. Tracked as
+  TRTLLM-14904.
+- FP8 KV cache (`kv_cache_config.dtype: fp8`) is not yet supported.
+- Speculative decoding is not yet supported.
+- Disaggregated serving is not yet supported.

examples/kimi_k3/eval_extra_llm_options.yaml

@@ -0,0 +1,19 @@
+tensor_parallel_size: 16
+enable_attention_dp: true
+moe_expert_parallel_size: 16
+max_batch_size: 32
+max_num_tokens: 8192
+max_seq_len: 8192
+trust_remote_code: true
+disable_overlap_scheduler: false
+enable_chunked_prefill: true
+cuda_graph_config:
+  enable_padding: true
+  max_batch_size: 32
+moe_config:
+  max_num_tokens: 33024
+  use_low_precision_moe_combine: true
+kv_cache_config:
+  enable_block_reuse: false
+  free_gpu_memory_fraction: 0.25
+  tokens_per_block: 64

examples/kimi_k3/eval_extra_llm_options_reuse.yaml

@@ -0,0 +1,30 @@
+# Identical to eval_extra_llm_options.yaml except for the reuse-specific
+# keys: kv_cache_config.enable_block_reuse is flipped to true and
+# mamba_state_config.periodic_snapshot_interval is set. Keep every other
+# key in sync with the base file when editing either.
+tensor_parallel_size: 16
+enable_attention_dp: true
+moe_expert_parallel_size: 16
+max_batch_size: 32
+max_num_tokens: 8192
+max_seq_len: 8192
+trust_remote_code: true
+disable_overlap_scheduler: false
+enable_chunked_prefill: true
+cuda_graph_config:
+  enable_padding: true
+  max_batch_size: 32
+moe_config:
+  max_num_tokens: 33024
+  use_low_precision_moe_combine: true
+kv_cache_config:
+  enable_block_reuse: true
+  free_gpu_memory_fraction: 0.25
+  tokens_per_block: 64
+  # Restore the pre-rework snapshot cadence: upstream deprecated
+  # mamba_state_cache_interval (default 256) into
+  # mamba_state_config.periodic_snapshot_interval (default 0 = snapshots
+  # off), and hybrid models only expose reusable prefixes at snapshot
+  # boundaries — without this, block reuse silently never engages.
+  mamba_state_config:
+    periodic_snapshot_interval: 256