NullSense commited on
Commit
46fae94
·
verified ·
1 Parent(s): 6a4f3cb

Qualify spec recommendation: eagle3 for short-context, ngram for long-document summarize (measured)

Browse files
Files changed (1) hide show
  1. README.md +6 -4
README.md CHANGED
@@ -148,10 +148,12 @@ vllm serve <this-repo> --trust-remote-code \
148
 
149
  Since 2026-07-23 the recommended speculation is my
150
  [EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3):
151
- 221 freeform / 244 summarize / 262 code tok/s in one config (TRITON_ATTN required
152
- with it; FlashInfer + spec drops CUDA graphs to piecewise on SM120). The previous
153
- ngram config (`{"method":"ngram","num_speculative_tokens":8,"prompt_lookup_max":4,
154
- "prompt_lookup_min":2}`) remains the head-less alternative for copy-heavy work.
 
 
155
 
156
  - Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}`
157
  (+ `preserve_thinking: false` for chat) to disable. Sampling defaults
 
148
 
149
  Since 2026-07-23 the recommended speculation is my
150
  [EAGLE-3 draft head](https://huggingface.co/NullSense/Nanbeige4.2-3B-EAGLE3):
151
+ 221 freeform / 244 short-passage summarize / 262 code tok/s in one config
152
+ (TRITON_ATTN required with it; FlashInfer + spec drops CUDA graphs to piecewise on
153
+ SM120). One measured exception: LONG-document summarization (multi-k-char inputs)
154
+ still favors the ngram config (`{"method":"ngram","num_speculative_tokens":8,
155
+ "prompt_lookup_max":4,"prompt_lookup_min":2}`): 231 vs 160 output tok/s on 16 real
156
+ articles; verbatim copy-spans reward lookup drafting. Pick per dominant workload.
157
 
158
  - Thinking ON by default; pass `chat_template_kwargs: {"enable_thinking": false}`
159
  (+ `preserve_thinking: false` for chat) to disable. Sampling defaults