Add Kimi-K2.7-Code Eagle3-MLA draft (full vocab) + README with per-benchmark accept_length

README.md

@@ -0,0 +1,88 @@
+---
+license: mit
+base_model: moonshotai/Kimi-K2.7-Code
+tags:
+- text-generation
+- speculative-decoding
+- eagle3
+- eagle3-mla
+- draft-model
+- vllm
+language:
+- en
+---
+
+# kimi-k2.7-code-eagle3-mla
+
+## Model Overview
+
+kimi-k2.7-code-eagle3-mla is an Eagle3 MTP draft model with MLA (Multi-Latent Attention) for
+accelerating inference of **Kimi-K2.7-Code** under vLLM speculative decoding. The draft proposes
+`num_speculative_tokens` candidate tokens per step; the Kimi-K2.7-Code verifier accepts them in
+parallel, so the output distribution is identical to plain autoregressive decoding while decode
+throughput improves.
+
+### Why an MLA (Multi-Latent Attention) Draft Model
+
+Compared with an MHA draft model, the MLA variant is a better fit for Kimi-K2.7-Code deployment:
+
+- Uses less KV cache, which reduces serving memory pressure.
+- Matches Kimi-K2.7-Code's MLA architecture, so it fits more naturally into the inference engine's
+  KV-cache handling under different serving scenarios such as PD-Disaggregation.
+
+### Architecture
+
+- **Algorithm**: EAGLE-3 with MLA, single draft decoder layer.
+- **Verifier**: Kimi-K2.7-Code. The draft reuses the verifier's frozen embedding / lm_head / norm
+  and trains one MLA decoder layer plus an auxiliary-hidden-state fusion layer.
+- **Draft vocabulary**: full 163,840-token vocabulary (no truncation).
+
+### Training Setup
+
+- **Framework**: **Camelot**, an online speculative-decoding training framework — FSDP training
+  and vLLM inference run concurrently, with the verifier continuously generating fresh training
+  data.
+- **Training data**: Kimi-K2.7-Code native data (agentic / coding / tool trajectories and
+  re-answered prompts).
+- **Schedule**: cosine LR 2e-5, sequence length 8192, `ttt_steps=4`.
+
+## Performance
+
+The primary metric is **accept_length** — the average number of tokens accepted per speculation
+step with `num_speculative_tokens=3`. Higher is better.
+
+Benchmarks were run on vLLM 0.20.0 (TP=8, greedy decoding, concurrency=1) against the
+Kimi-K2.7-Code verifier.
+
+| Category | Benchmark | N | Accept Length |
+| --- | --- | --- | --- |
+| Dialogue | [MTBench](https://github.com/lm-sys/FastChat/tree/main/fastchat/llm_judge) | 80 | 2.427 |
+| Chinese | [CEval](https://huggingface.co/datasets/ceval/ceval-exam) | 212 | 2.348 |
+| Math | [GSM8K](https://github.com/openai/grade-school-math) | 500 | 3.201 |
+| Code | [HumanEval](https://huggingface.co/datasets/openai/openai_humaneval) | 164 | 2.738 |
+| Math | [MATH500](https://huggingface.co/datasets/HuggingFaceH4/MATH-500) | 500 | 2.918 |
+| Math | [AIME](https://huggingface.co/datasets/Maxwell-Jia/AIME_2024) | 30 | 2.542 |
+| Code | [LiveCodeBench](https://huggingface.co/datasets/livecodebench/code_generation) | 200 | 2.362 |
+| Code | [SPEED-Bench (coding)](https://huggingface.co/datasets/nvidia/SPEED-Bench) | 80 | 2.515 |
+
+---
+
+## Quick Start
+
+### Requirements
+
+- NVIDIA GPU with CUDA 12.0+
+- [vLLM](https://github.com/vllm-project/vllm) >= 0.20.0
+
+### Launch Server (vLLM)
+
+```bash
+vllm serve moonshotai/Kimi-K2.7-Code \
+    --tensor-parallel-size 8 \
+    --speculative-config '{"model": "novita/kimi-k2.7-code-eagle3-mla", "method": "eagle3", "num_speculative_tokens": 3}' \
+    --trust-remote-code
+```
+
+### Launch Server (SGLang)
+
+MLA Eagle3 draft model is not yet supported in SGLang. Will update once support is available.

config.json

@@ -0,0 +1,47 @@
+{
+  "architectures": [
+    "Eagle3DeepseekV2ForCausalLM"
+  ],
+  "model_type": "kimi_k2",
+  "hidden_size": 7168,
+  "intermediate_size": 18432,
+  "num_hidden_layers": 1,
+  "num_attention_heads": 64,
+  "num_key_value_heads": 64,
+  "q_lora_rank": 1536,
+  "kv_lora_rank": 512,
+  "qk_nope_head_dim": 128,
+  "qk_rope_head_dim": 64,
+  "v_head_dim": 128,
+  "hidden_act": "silu",
+  "rms_norm_eps": 1e-05,
+  "vocab_size": 163840,
+  "draft_vocab_size": 163840,
+  "torch_dtype": "bfloat16",
+  "rope_theta": 50000.0,
+  "rope_scaling": {
+    "beta_fast": 1.0,
+    "beta_slow": 1.0,
+    "factor": 64.0,
+    "mscale": 1.0,
+    "mscale_all_dim": 1.0,
+    "original_max_position_embeddings": 4096,
+    "type": "yarn"
+  },
+  "eagle_config": {
+    "eagle_aux_hidden_state_layer_ids": [
+      2,
+      30,
+      58
+    ],
+    "use_aux_hidden_state": true,
+    "use_input_layernorm_in_first_layer": true,
+    "use_last_layernorm": true,
+    "use_mtp_layernorm": false
+  },
+  "bos_token_id": 163584,
+  "eos_token_id": 163585,
+  "pad_token_id": 0,
+  "_torchspec_version": "0.1.0",
+  "max_position_embeddings": 262144
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd5a71a1027bda3116df4d1abaecc5ad6c2e1d25009508ce08924e0f15b85d2b
+size 6031210296