| --- |
| license: other |
| --- |
| |
| <!-- Draft: review the runtime requirements and benchmark wording before publishing. --> |
|
|
| # Kimi-K2.7 Coder Eagle3.1 MLA |
|
|
| EAGLE3 draft model for speculative decoding with [Kimi-K2.7-Code](https://modelscope.cn/models/moonshotai/Kimi-K2.7-Code). |
|
|
| Continued training from [kimi-k2.6-eagle3.1-mla](https://modelscope.cn/models/lightseekorg/kimi-k2.6-eagle3.1-mla). |
|
|
| Co-authored by [KVCache.AI](https://kvcache.ai/) and LightSeek. |
|
|
| This checkpoint uses MLA and a one-layer EAGLE3.1 draft architecture with `fc_norm` and `norm_output`. |
|
|
| ## Features |
|
|
| - **EAGLE3.1 + MLA**: One-layer draft model using auxiliary hidden states from target layers 1, 29, and 57. |
| - **fc_norm**: Per-chunk RMSNorm on auxiliary hidden states before FC projection. |
| - **norm_output**: Uses post-norm hidden states as auxiliary output. |
| - **Long-context configuration**: YaRN RoPE configuration with a maximum position length of 262,144 tokens. |
|
|
| ## Benchmark Results |
|
|
| Target model: [moonshotai/Kimi-K2.7-Code](https://modelscope.cn/models/moonshotai/Kimi-K2.7-Code) |
|
|
| Draft model: [lightseekorg/kimi-k2.7-coder-eagle3.1-mla](https://huggingface.co/lightseekorg/kimi-k2.7-coder-eagle3.1-mla) |
|
|
| ### 3-token draft |
|
|
| The run used SGLang 0.5.14, TP=8, EP=1, BF16, `topk=1`, `num_steps=3`, `num_draft_tokens=4`. |
|
|
| | Benchmark | Samples | AccLen | AccRate | |
| | --- | ---: | ---: | ---: | |
| | MTBench | 80 | 2.345209 | 44.838445% | |
| | CEval | 212 | 2.051672 | 35.075910% | |
| | GSM8K | 500 | 2.946559 | 65.499844% | |
| | HumanEval | 164 | 2.544423 | 51.495064% | |
| | MATH500 | 500 | 2.664324 | 55.479869% | |
| | AIME | 30 | 2.431353 | 47.711782% | |
| | MMStar | 200 | 2.305465 | 43.584233% | |
|
|
| ## Usage with vLLM |
|
|
| https://recipes.vllm.ai/moonshotai/Kimi-K2.7-Code |
|
|
| ## Usage with SGLang |
|
|
| <!-- TODO: Replace the target model path and verify the minimum supported SGLang version before publishing. --> |
|
|
| ```bash |
| |
| python3 -m sglang.launch_server \ |
| --model-path /path/to/Kimi-K2.7-Code \ |
| --tp 8 --ep-size 1 --trust-remote-code \ |
| --speculative-algorithm EAGLE3 \ |
| --speculative-draft-model-path lightseekorg/kimi-k2.7-coder-eagle3.1-mla \ |
| --speculative-num-steps 3 \ |
| --speculative-eagle-topk 1 \ |
| --speculative-num-draft-tokens 4 \ |
| --decode-attention-backend flashinfer \ |
| --speculative-draft-attention-backend flashinfer \ |
| --mem-fraction-static 0.75 \ |
| --page-size 1 \ |
| --max-running-requests 8 \ |
| --dtype bfloat16 \ |
| --host 0.0.0.0 \ |
| --port 30000 \ |
| --reasoning-parser kimi_k2 \ |
| --tool-call-parser kimi_k2 \ |
| --cuda-graph-backend-decode full \ |
| --cuda-graph-max-bs-decode 1 \ |
| --cuda-graph-bs-decode 1 \ |
| ``` |
