| --- |
| pipeline_tag: text-generation |
| base_model: moonshotai/Kimi-K3 |
| checkpoint_step: 5780 |
| tags: |
| - speculative-decoding |
| - vllm |
| - dspark |
| - kimi-k3 |
| --- |
| |
| # Kimi-K3 DSpark — Block 5 |
|
|
| A DSpark draft model for speculative decoding with |
| [`moonshotai/Kimi-K3`](https://huggingface.co/moonshotai/Kimi-K3). Three full-causal MLA |
| layers, conditioned on hidden states extracted from three of the target's layers and |
| trained at `block_size=5`: one forward pass drafts the full five-token block. |
|
|
| ## Training |
|
|
| The draft completed a four-epoch run with |
| [**speculators**](https://github.com/vllm-project/speculators), the speculative-decoding |
| library from the [vLLM project](https://github.com/vllm-project), against Kimi-K3 hidden |
| states streamed live from **vLLM**. |
|
|
| ## Acceptance |
|
|
| Evaluated against Kimi-K3 on 8×B300 (TP8), drafting 5 tokens with probabilistic |
| sampling and block rejection at temperature 1.0 and top-p 0.95. All 1,604 requests |
| succeeded. The aggregate is computed from summed counters, not an average of arm rates. |
|
|
| | Workload | Requests | Acceptance | Mean accepted length (max 6) | |
| | --- | ---: | ---: | ---: | |
| | GSM8K | 256 | 71.62% | 4.581 | |
| | HumanEval | 164 | 63.74% | 4.187 | |
| | MBPP | 256 | 56.38% | 3.819 | |
| | MATH-500 | 500 | 47.48% | 3.374 | |
| | SWE-bench Pro | 128 | 38.09% | 2.904 | |
| | MT-Bench | 80 | 33.19% | 2.659 | |
| | AIME26 | 30 | 29.78% | 2.489 | |
| | AA-LCR (~100K) | 100 | 41.04% | 3.052 | |
| | BEAM 100K | 20 | 37.17% | 2.858 | |
| | BEAM 500K | 35 | 30.26% | 2.513 | |
| | BEAM 1M | 35 | 31.37% | 2.569 | |
| | **Aggregate** | **1,604** | **46.10%** | **3.305** | |
|
|
| The separate five-shot GSM8K accuracy check scored **96.51% exact match** on all 1,319 |
| examples with zero request errors. |
|
|
| **The full 1M context window is supported through YaRN**, evaluated on every row of the |
| [BEAM](https://huggingface.co/datasets/Mohammadta/BEAM) 100K, 500K, and 1M splits above. |
| The fixed 1M acceptance corpus left-truncates oldest turns in 14 of 35 over-limit source |
| conversations to keep prompt plus output within 1,048,576 tokens; its longest measured |
| prompt is 1,039,871 tokens. Two successful completions were flagged as degenerate by the |
| client (BEAM 500K conversation 4 and BEAM 1M conversation 18). Full counters, |
| per-position rates, and quality flags are in `benchmark_results.json`. |
|
|
| ## Serving |
|
|
| ```bash |
| vllm serve moonshotai/Kimi-K3 \ |
| --trust-remote-code \ |
| --tensor-parallel-size 8 \ |
| --max-model-len 1048576 \ |
| --speculative-config '{ |
| "method": "dspark", |
| "model": "Inferact/Kimi-K3-DSpark-Block5", |
| "num_speculative_tokens": 5, |
| "attention_backend": "FLASHINFER_MLA", |
| "draft_sample_method": "probabilistic", |
| "rejection_sample_method": "block" |
| }' |
| ``` |
|
|
