Add verified B300 evaluation at a 4,096-token output cap
#2
by shanjiaz - opened
README.md
CHANGED
|
@@ -35,27 +35,68 @@ the context grows. This keeps drafting efficient at long context, up to Kimi K3.
|
|
| 35 |
- **Client:** chat completions, temperature 1.0, top_p 0.95, full test set per benchmark.
|
| 36 |
- **Metric:** acceptance length = 1 + accepted draft tokens / drafts (counts the bonus token).
|
| 37 |
|
| 38 |
-
##
|
| 39 |
-
|
| 40 |
-
|
| 41 |
-
|
| 42 |
-
|
| 43 |
-
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
|
| 47 |
-
|
| 48 |
-
|
| 49 |
-
|
| 50 |
-
|
| 51 |
-
|
| 52 |
-
|
| 53 |
-
|
| 54 |
-
|
| 55 |
-
|
| 56 |
-
|
| 57 |
-
|
| 58 |
-
|
| 59 |
-
|
| 60 |
-
|
| 61 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
- **Client:** chat completions, temperature 1.0, top_p 0.95, full test set per benchmark.
|
| 36 |
- **Metric:** acceptance length = 1 + accepted draft tokens / drafts (counts the bonus token).
|
| 37 |
|
| 38 |
+
## Serving with vLLM
|
| 39 |
+
|
| 40 |
+
```bash
|
| 41 |
+
VLLM_KIMI_K3_AUX_ATTN_RES_STREAM=0 \
|
| 42 |
+
vllm serve moonshotai/Kimi-K3.1-Pro \
|
| 43 |
+
--host 127.0.0.1 --port 19142 \
|
| 44 |
+
--trust-remote-code \
|
| 45 |
+
--tensor-parallel-size 8 \
|
| 46 |
+
--load-format fastsafetensors \
|
| 47 |
+
--reasoning-parser kimi_k3 \
|
| 48 |
+
--max-model-len 65536 \
|
| 49 |
+
--max-num-seqs 128 \
|
| 50 |
+
--max-num-batched-tokens 8192 \
|
| 51 |
+
--gpu-memory-utilization 0.90 \
|
| 52 |
+
--no-enable-prefix-caching \
|
| 53 |
+
--per-request-spec-decode-metrics summary \
|
| 54 |
+
--speculative-config '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
## Per-position draft acceptance
|
| 58 |
+
|
| 59 |
+
Positions 1–7 refer to the proposed draft tokens. Rates are percentages of verification steps.
|
| 60 |
+
|
| 61 |
+
| Benchmark | Prompts | Max tokens | Acceptance length | Position 1 | Position 2 | Position 3 | Position 4 | Position 5 | Position 6 | Position 7 |
|
| 62 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 63 |
+
| GSM8K | 1,319 | 1,024 | **5.566** | 91% | 82% | 72% | 64% | 56% | 49% | 43% |
|
| 64 |
+
| HumanEval | 164 | 1,024 | 4.660 | 86% | 71% | 59% | 49% | 40% | 34% | 28% |
|
| 65 |
+
| MBPP | 974 | 1,024 | 4.376 | 84% | 68% | 55% | 44% | 35% | 28% | 23% |
|
| 66 |
+
| MATH-500 | 500 | 2,048 | 4.445 | 84% | 69% | 56% | 45% | 37% | 30% | 24% |
|
| 67 |
+
| SPEED-Bench low entropy | 302 | 1,536 | 3.619 | 78% | 58% | 42% | 31% | 23% | 17% | 13% |
|
| 68 |
+
| SPEED-Bench coding | 80 | 4,096 | 4.159 | 81% | 64% | 50% | 40% | 32% | 27% | 22% |
|
| 69 |
+
| MT-Bench | 80 | 1,024 | 3.393 | 76% | 54% | 38% | 27% | 20% | 14% | 11% |
|
| 70 |
+
| SWE-bench Pro | 642 | 2,048 | 3.302 | 75% | 53% | 37% | 26% | 18% | 13% | 9% |
|
| 71 |
+
| AIME 2026 | 30 | 4,096 | 3.429 | 77% | 56% | 40% | 28% | 20% | 14% | 10% |
|
| 72 |
+
| **Pooled** | — | — | **3.864** | **79%** | **61%** | **46%** | **35%** | **27%** | **21%** | **17%** |
|
| 73 |
+
|
| 74 |
+
## SpeedBench acceptance
|
| 75 |
+
|
| 76 |
+
| Category | Conversations | Model calls | Max tokens | Acceptance length | Draft acceptance |
|
| 77 |
+
| --- | ---: | ---: | ---: | ---: | ---: |
|
| 78 |
+
| Coding | 80 | 89 | 4,096 | 4.0957 | 44.22% |
|
| 79 |
+
| Humanities | 80 | 88 | 4,096 | 3.0354 | 29.08% |
|
| 80 |
+
| Math | 80 | 98 | 4,096 | 3.2025 | 31.46% |
|
| 81 |
+
| QA | 80 | 80 | 4,096 | 3.4800 | 35.43% |
|
| 82 |
+
| RAG | 80 | 105 | 4,096 | 4.2890 | 46.99% |
|
| 83 |
+
| Reasoning | 80 | 190 | 4,096 | 3.5954 | 37.08% |
|
| 84 |
+
| STEM | 80 | 86 | 4,096 | 3.1874 | 31.25% |
|
| 85 |
+
| Writing | 80 | 84 | 4,096 | 2.8243 | 26.06% |
|
| 86 |
+
| Multilingual | 80 | 80 | 4,096 | 4.0903 | 44.15% |
|
| 87 |
+
| Summarization | 80 | 80 | 4,096 | 4.1758 | 45.37% |
|
| 88 |
+
| Roleplay | 80 | 158 | 4,096 | 3.0310 | 29.01% |
|
| 89 |
+
| **Pooled** | **880** | **1,138** | **4,096** | **3.2690** | **32.41%** |
|
| 90 |
+
|
| 91 |
+
## Long context acceptance
|
| 92 |
+
|
| 93 |
+
| Benchmark | Prompts | Prompt tokens | Max tokens | Acceptance length | Position 1 | Position 2 | Position 3 | Position 4 | Position 5 | Position 6 | Position 7 |
|
| 94 |
+
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
| 95 |
+
| AA-LCR | 100 | 95K mean | 1,024 | **3.935** | 81% | 64% | 49% | 37% | 27% | 21% | 15% |
|
| 96 |
+
| BEAM 128K | 400 | 97K–125K | 2,048 | 3.516 | 78% | 58% | 41% | 29% | 20% | 14% | 10% |
|
| 97 |
+
| BEAM 1M | 700 | 826K–1.04M | 2,048 | 3.421 | 77% | 56% | 40% | 28% | 19% | 13% | 9% |
|
| 98 |
+
|
| 99 |
+
### Throughput and interactivity
|
| 100 |
+
|
| 101 |
+

|
| 102 |
+
|