Qualify spec recommendation: eagle3 for short-context, ngram for long-document summarize (measured)
Browse files
README.md
CHANGED
|
@@ -148,10 +148,12 @@ vllm serve <this-repo> --trust-remote-code \
|
|
| 148 |
|
| 149 |
Since 2026-07-23 the recommended speculation is my
|
| 150 |
[EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3):
|
| 151 |
-
221 freeform / 244 summarize / 262 code tok/s in one config
|
| 152 |
-
with it; FlashInfer + spec drops CUDA graphs to piecewise on
|
| 153 |
-
|
| 154 |
-
|
|
|
|
|
|
|
| 155 |
|
| 156 |
- Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}`
|
| 157 |
(+ `preserve_thinking: false` for chat) to disable. Sampling defaults
|
|
|
|
| 148 |
|
| 149 |
Since 2026-07-23 the recommended speculation is my
|
| 150 |
[EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3):
|
| 151 |
+
221 freeform / 244 short-passage summarize / 262 code tok/s in one config
|
| 152 |
+
(TRITON_ATTN required with it; FlashInfer + spec drops CUDA graphs to piecewise on
|
| 153 |
+
SM120). One measured exception: LONG-document summarization (multi-k-char inputs)
|
| 154 |
+
still favors the ngram config (`{"method":"ngram","num_speculative_tokens":8,
|
| 155 |
+
"prompt_lookup_max":4,"prompt_lookup_min":2}`): 231 vs 160 output tok/s on 16 real
|
| 156 |
+
articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload.
|
| 157 |
|
| 158 |
- Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}`
|
| 159 |
(+ `preserve_thinking: false` for chat) to disable. Sampling defaults
|