| --- |
| license: other |
| base_model: moonshotai/Kimi-K2.6 |
| tags: |
| - text-generation |
| - speculative-decoding |
| - eagle3 |
| - kimi-k2.6 |
| - mla |
| - torchspec |
| --- |
| |
| # kimi-k2.6-eagle3-mla |
|
|
| Eagle3 MTP draft model with MLA (Multi-Latent Attention) for accelerating |
| inference of [Kimi-K2.6](https://huggingface.co/moonshotai/Kimi-K2.6). |
|
|
| This is a fine-tuned draft, anchored to the official |
| [lightseekorg/kimi-k2.6-eagle3-mla](https://huggingface.co/lightseekorg/kimi-k2.6-eagle3-mla) |
| initialization. It targets multi-hop (downstream-position) acceptance while |
| preserving the first-hop gain, evaluated by runtime accept-length on a frozen |
| full-context held-out set. |
|
|
| ## Fine-tune setup |
|
|
| - **Init**: lightseekorg/kimi-k2.6-eagle3-mla (official MLA weights) |
| - **Objective**: Eagle3 distillation + multi-step TTT supervision |
| (`ttt_steps=4`, `ttt_step_loss_decay=1.0`, off-policy downstream tokens) |
| - **Anti-over-specialization**: L2-SP weight-space anchor toward the init |
| (penalize trainable-param drift; lambda=1e-4) |
| - **Optimizer**: lr 2e-5, cosine schedule |
| - **Checkpoint**: best by held-out validation loss on the K2.6 ruler |
| (step 95400; val_loss 5.490, the global minimum of the v3 run) |
| |
| ## Performance |
| |
| Primary metric is **accept_length** — average tokens accepted per speculation |
| step with `num_speculative_tokens=3` (higher is better). Evaluated with |
| vLLM 0.20.0 on 8x H200, TP=8, max-model-len 32768, greedy. |
| |
| On a frozen K2.6 full-context held-out judge set (914 prompts): |
| |
| | Model | accept_len | |
| |-------|-----------:| |
| | lightseek (official init) | 2.285 | |
| | this model | **2.308** | |
|
|
| This draft improves over the official init on the K2.6 held-out distribution. |
|
|
| ## Note on distribution shift |
|
|
| This checkpoint is selected by validation loss on the K2.6 teacher |
| distribution. In cross-version testing against real Kimi-K2.7-Code production |
| traffic, the official lightseek init currently shows higher accept-length than |
| this fine-tune — i.e. the K2.6 fine-tune over-specializes to its training |
| distribution. If your serving traffic differs substantially from long |
| multi-turn K2.6 dialogues, benchmark both this draft and the lightseek init on |
| your own traffic before choosing. (The L2-SP anchor above is intended to |
| mitigate this; tuning it against real-traffic accept-length is ongoing.) |
|
|
| ## Usage |
|
|
| Serve with vLLM as the speculative draft for Kimi-K2.6, with |
| `num_speculative_tokens=3` in the speculative-config. |
|
|
