| --- |
| license: other |
| license_name: modified-mit |
| license_link: https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/LICENSE |
| base_model: moonshotai/Kimi-K2.5 |
| tags: |
| - mtp |
| - speculative-decoding |
| - kimi |
| - deepseek |
| - moe |
| --- |
| |
| # Kimi-K2.5-MTP |
|
|
| Kimi K2.5 with Multi-Token Prediction (MTP) speculative decoding support for vLLM. |
|
|
| ## What is this? |
|
|
| This model combines: |
| - **Base model**: [moonshotai/Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) (1.04T/32B active MoE, INT4 compressed-tensors) |
| - **MTP layer**: Layer 61 MTP weights from [k-l-lambda/Kimi-K2.5-MTP](https://huggingface.co/k-l-lambda/Kimi-K2.5-MTP) (14.7 GB, INT4 compressed-tensors for experts, BF16 for projections/norms/head) |
|
|
| The MTP module enables speculative decoding where a lightweight prediction head drafts tokens that the main model can verify in parallel, improving throughput. |
|
|
| ## Usage with vLLM |
|
|
| Requires [modularml/vllm](https://github.com/modularml/vllm) with MTP support for Kimi K2.5 (branch `kimi-k25-mtp-780ba37`). |
|
|
| ```bash |
| vllm serve modularai/Kimi-K2.5-MTP \ |
| --tensor-parallel-size 8 \ |
| --trust-remote-code \ |
| --speculative-config '{"model":"modularai/Kimi-K2.5-MTP","method":"mtp","num_speculative_tokens":1,"use_local_argmax_reduction":true}' |
| ``` |
|
|
| ## Benchmark Results (8x NVIDIA B200) |
|
|
| | Config | Output tok/s | TPOT p50 (ms) | Acceptance Rate | |
| |--------|-------------|---------------|-----------------| |
| | No speculation | 947 | 14.07 | — | |
| | MTP k=1 | 869 | 15.85 | ~39% | |
|
|
| > **Note**: The MTP acceptance rate is low (~39%) because the MTP weights were not trained directly on this base model checkpoint. With properly matched MTP weights (trained via self-distillation on this exact checkpoint), acceptance rates of 80-90% are expected, yielding ~1.5-1.8x throughput improvement. |
|
|
| ## Architecture |
|
|
| - **Model type**: `kimi_k25` (VLM wrapper around DeepSeek V3 architecture) |
| - **Parameters**: 1.04T total / 32B active (MoE with 384 routed experts) |
| - **MTP module**: 1 full decoder layer at index 61 (~14B parameters) |
| - `enorm` + `hnorm` (RMSNorm) → concat → `eh_proj` (Linear 2×7168 → 7168) → decoder layer (MLA attention + MoE) → `shared_head` (RMSNorm + LM head) |
| - **Quantization**: INT4 compressed-tensors (group_size=32) for MoE experts, BF16 for dense layers |
| |
| ## License |
| |
| This model uses weights from [moonshotai/Kimi-K2.5](https://huggingface.co/moonshotai/Kimi-K2.5) under the [Modified MIT License](https://huggingface.co/moonshotai/Kimi-K2.5/blob/main/LICENSE) and MTP weights from [k-l-lambda/Kimi-K2.5-MTP](https://huggingface.co/k-l-lambda/Kimi-K2.5-MTP) under MIT. |
| |
| ## Credits |
| |
| - Base model: [Moonshot AI](https://huggingface.co/moonshotai) |
| - MTP weights: [k-l-lambda](https://huggingface.co/k-l-lambda) |
| |
