Add verified B300 evaluation at a 4,096-token output cap

#2
by shanjiaz - opened
Files changed (1) hide show
  1. README.md +65 -24
README.md CHANGED
@@ -35,27 +35,68 @@ the context grows. This keeps drafting efficient at long context, up to Kimi K3.
35
  - **Client:** chat completions, temperature 1.0, top_p 0.95, full test set per benchmark.
36
  - **Metric:** acceptance length = 1 + accepted draft tokens / drafts (counts the bonus token).
37
 
38
- ## Results
39
-
40
- <!-- Fill in from the final checkpoint's evaluation before release. -->
41
-
42
- | Benchmark | Prompts | Max tokens | Acceptance length | Accept rate, positions 1-7 |
43
- | --- | ---: | ---: | ---: | --- |
44
- | GSM8K | 1,319 | 1,024 | **5.566** | 0.91 0.82 0.72 0.64 0.56 0.49 0.43 |
45
- | HumanEval | 164 | 1,024 | 4.660 | 0.86 0.71 0.59 0.49 0.40 0.34 0.28 |
46
- | MBPP | 974 | 1,024 | 4.376 | 0.84 0.68 0.55 0.44 0.35 0.28 0.23 |
47
- | MATH-500 | 500 | 2,048 | 4.445 | 0.84 0.69 0.56 0.45 0.37 0.30 0.24 |
48
- | SPEED-Bench low entropy (10K-token inputs) | 302 | 1,536 | 3.619 | 0.78 0.58 0.42 0.31 0.23 0.17 0.13 |
49
- | SPEED-Bench coding | 80 | 4,096 | 4.159 | 0.81 0.64 0.50 0.40 0.32 0.27 0.22 |
50
- | MT-Bench | 80 | 1,024 | 3.393 | 0.76 0.54 0.38 0.27 0.20 0.14 0.11 |
51
- | SWE-bench Pro | 642 | 2,048 | 3.302 | 0.75 0.53 0.37 0.26 0.18 0.13 0.09 |
52
- | AIME 2026 | 30 | 4,096 | 3.429 | 0.77 0.56 0.40 0.28 0.20 0.14 0.10 |
53
- | **Pooled** | | | **3.864** | 0.79 0.61 0.46 0.35 0.27 0.21 0.17 |
54
-
55
- ## Long context
56
-
57
- | Benchmark | Prompts | Prompt tokens | Max tokens | Acceptance length | Accept rate, positions 1-7 |
58
- | --- | ---: | ---: | ---: | ---: | --- |
59
- | AA-LCR | 100 | 95K mean | 1,024 | **3.935** | 0.81 0.64 0.49 0.37 0.27 0.21 0.15 |
60
- | BEAM 128K | 400 | 97K-125K | 2,048 | 3.516 | 0.78 0.58 0.41 0.29 0.20 0.14 0.10 |
61
- | BEAM 1M | 700 | 826K-1.04M | 2,048 | 3.421 | 0.77 0.56 0.40 0.28 0.19 0.13 0.09 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  - **Client:** chat completions, temperature 1.0, top_p 0.95, full test set per benchmark.
36
  - **Metric:** acceptance length = 1 + accepted draft tokens / drafts (counts the bonus token).
37
 
38
+ ## Serving with vLLM
39
+
40
+ ```bash
41
+ VLLM_KIMI_K3_AUX_ATTN_RES_STREAM=0 \
42
+ vllm serve moonshotai/Kimi-K3.1-Pro \
43
+ --host 127.0.0.1 --port 19142 \
44
+ --trust-remote-code \
45
+ --tensor-parallel-size 8 \
46
+ --load-format fastsafetensors \
47
+ --reasoning-parser kimi_k3 \
48
+ --max-model-len 65536 \
49
+ --max-num-seqs 128 \
50
+ --max-num-batched-tokens 8192 \
51
+ --gpu-memory-utilization 0.90 \
52
+ --no-enable-prefix-caching \
53
+ --per-request-spec-decode-metrics summary \
54
+ --speculative-config '{"method":"dspark","model":"Inferact/Kimi-K3.1-Pro-DSpark","num_speculative_tokens":7,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
55
+ ```
56
+
57
+ ## Per-position draft acceptance
58
+
59
+ Positions 1–7 refer to the proposed draft tokens. Rates are percentages of verification steps.
60
+
61
+ | Benchmark | Prompts | Max tokens | Acceptance length | Position 1 | Position 2 | Position 3 | Position 4 | Position 5 | Position 6 | Position 7 |
62
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
63
+ | GSM8K | 1,319 | 1,024 | **5.566** | 91% | 82% | 72% | 64% | 56% | 49% | 43% |
64
+ | HumanEval | 164 | 1,024 | 4.660 | 86% | 71% | 59% | 49% | 40% | 34% | 28% |
65
+ | MBPP | 974 | 1,024 | 4.376 | 84% | 68% | 55% | 44% | 35% | 28% | 23% |
66
+ | MATH-500 | 500 | 2,048 | 4.445 | 84% | 69% | 56% | 45% | 37% | 30% | 24% |
67
+ | SPEED-Bench low entropy | 302 | 1,536 | 3.619 | 78% | 58% | 42% | 31% | 23% | 17% | 13% |
68
+ | SPEED-Bench coding | 80 | 4,096 | 4.159 | 81% | 64% | 50% | 40% | 32% | 27% | 22% |
69
+ | MT-Bench | 80 | 1,024 | 3.393 | 76% | 54% | 38% | 27% | 20% | 14% | 11% |
70
+ | SWE-bench Pro | 642 | 2,048 | 3.302 | 75% | 53% | 37% | 26% | 18% | 13% | 9% |
71
+ | AIME 2026 | 30 | 4,096 | 3.429 | 77% | 56% | 40% | 28% | 20% | 14% | 10% |
72
+ | **Pooled** | — | — | **3.864** | **79%** | **61%** | **46%** | **35%** | **27%** | **21%** | **17%** |
73
+
74
+ ## SpeedBench acceptance
75
+
76
+ | Category | Conversations | Model calls | Max tokens | Acceptance length | Draft acceptance |
77
+ | --- | ---: | ---: | ---: | ---: | ---: |
78
+ | Coding | 80 | 89 | 4,096 | 4.0957 | 44.22% |
79
+ | Humanities | 80 | 88 | 4,096 | 3.0354 | 29.08% |
80
+ | Math | 80 | 98 | 4,096 | 3.2025 | 31.46% |
81
+ | QA | 80 | 80 | 4,096 | 3.4800 | 35.43% |
82
+ | RAG | 80 | 105 | 4,096 | 4.2890 | 46.99% |
83
+ | Reasoning | 80 | 190 | 4,096 | 3.5954 | 37.08% |
84
+ | STEM | 80 | 86 | 4,096 | 3.1874 | 31.25% |
85
+ | Writing | 80 | 84 | 4,096 | 2.8243 | 26.06% |
86
+ | Multilingual | 80 | 80 | 4,096 | 4.0903 | 44.15% |
87
+ | Summarization | 80 | 80 | 4,096 | 4.1758 | 45.37% |
88
+ | Roleplay | 80 | 158 | 4,096 | 3.0310 | 29.01% |
89
+ | **Pooled** | **880** | **1,138** | **4,096** | **3.2690** | **32.41%** |
90
+
91
+ ## Long context acceptance
92
+
93
+ | Benchmark | Prompts | Prompt tokens | Max tokens | Acceptance length | Position 1 | Position 2 | Position 3 | Position 4 | Position 5 | Position 6 | Position 7 |
94
+ | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
95
+ | AA-LCR | 100 | 95K mean | 1,024 | **3.935** | 81% | 64% | 49% | 37% | 27% | 21% | 15% |
96
+ | BEAM 128K | 400 | 97K–125K | 2,048 | 3.516 | 78% | 58% | 41% | 29% | 20% | 14% | 10% |
97
+ | BEAM 1M | 700 | 826K–1.04M | 2,048 | 3.421 | 77% | 56% | 40% | 28% | 19% | 13% | 9% |
98
+
99
+ ### Throughput and interactivity
100
+
101
+ ![image](https://cdn-uploads.huggingface.co/production/uploads/67f401f4bb5b52cdad90f9a7/ZmNQ856m60NFuGr-4ioRU.png)
102
+