Any MTP or DFlash/DSpark drafter planned?
Hey, does this model support MTP?
Also any plans to release a speculative decoding draft model (dflash / dspark)?
I’m training a DSpark one myself right now, will share once the training completes over the weekend.
we are supporting our model with our own speedup method: UNO, https://github.com/ifm-ai/uno
we have supported UNO for several of our models: https://huggingface.co/IFM/K2-Horizon-7B-Uno
K2-Horizon-3.7B-DSpark
Model Link: https://huggingface.co/rasyosef/K2-Horizon-3.7B-DSpark
A DSpark draft model for speculative decoding with IFM/K2-Horizon-3.7B as the verifier, trained with speculators. The drafter proposes 8 tokens at a time and the verifier checks them in one forward pass, so greedy output is identical to running the verifier alone — a lossless speedup. At temperature 0, mean acceptance length is 3.06 tokens committed per verification step, up to 4.86 on math_reasoning. Compared with the verifier alone, throughput is 2.72× higher on average, up to 4.12× on math_reasoning and 3.03× on HumanEval, and it is faster than the verifier alone on all nine benchmark subsets, by at least 2.16×.
Training code: rasyosef/train-dspark-draft-models.
Trained on 50,000 samples.
Evaluation
evaluate.py throughput across the nine RedHatAI/speculator_benchmarks subsets.
Setup
- Acceptance measured at two settings: temperature 0 (greedy decoding), and sampling with temperature 1.0, top_p 0.95 (the verifier's recommended settings)
- Throughput reported at temperature 0 only
- Hardware: single A100 40GB, max concurrency 1
- Same settings for the baseline and DSpark runs
acceptance_length is mean tokens committed per verification step, including the bonus token. Its floor is 1.0 and its ceiling is 9.0 at block size 8. Rows are in the same order in every table, sorted by acceptance length at temperature 0.
Acceptance at temperature 0
| subset | acceptance_length | pos_0 | pos_1 | pos_2 | pos_3 | pos_4 | pos_5 | pos_6 | pos_7 |
|---|---|---|---|---|---|---|---|---|---|
| math_reasoning | 4.857 | 85.5% | 71.4% | 59.5% | 48.8% | 40.0% | 32.6% | 26.7% | 21.2% |
| HumanEval | 3.420 | 76.1% | 54.1% | 38.0% | 26.4% | 18.6% | 13.0% | 9.2% | 6.6% |
| rag | 3.014 | 72.2% | 49.0% | 31.9% | 20.3% | 12.8% | 7.8% | 4.7% | 2.7% |
| writing | 2.925 | 67.4% | 43.8% | 28.8% | 19.6% | 13.5% | 9.1% | 6.2% | 4.1% |
| question | 2.905 | 66.8% | 43.2% | 28.5% | 19.4% | 13.3% | 9.0% | 6.1% | 4.2% |
| qa | 2.828 | 66.7% | 42.7% | 27.6% | 18.3% | 11.8% | 7.5% | 4.9% | 3.2% |
| tool_call | 2.803 | 69.5% | 44.7% | 28.0% | 16.6% | 10.1% | 5.8% | 3.5% | 2.0% |
| translation | 2.797 | 64.9% | 43.4% | 28.4% | 18.3% | 11.3% | 6.9% | 4.0% | 2.4% |
| summarization | 2.572 | 64.8% | 40.0% | 23.8% | 13.7% | 7.8% | 4.2% | 1.9% | 0.9% |
Throughput at temperature 0 (tokens/s)
Mean output throughput in tokens/s, using the setup above. Baseline is K2-Horizon-3.7B with no speculative decoding. The last column gives this model's throughput divided by baseline throughput for each subset.
| subset | baseline (no drafter) | DSpark (this model) | speedup |
|---|---|---|---|
| math_reasoning | 109.4 | 450.2 | 4.12× |
| HumanEval | 109.1 | 330.7 | 3.03× |
| rag | 108.1 | 265.1 | 2.45× |
| writing | 109.1 | 293.3 | 2.69× |
| question | 109.1 | 293.3 | 2.69× |
| qa | 108.3 | 275.8 | 2.55× |
| tool_call | 107.5 | 257.1 | 2.39× |
| translation | 109.5 | 268.2 | 2.45× |
| summarization | 108.0 | 233.2 | 2.16× |
| average speedup | 1.00× | 2.72× | 2.72× |
Usage
vLLM
vLLM loads the verifier automatically from the config — don't pass it separately. The verifier uses custom code, so --trust-remote-code is required, and K2-Horizon is a reasoning model, so enable its reasoning parser to get the thinking back in reasoning_content.
vllm serve rasyosef/K2-Horizon-3.7B-DSpark --port 8000 \
--trust-remote-code \
--reasoning-parser k2_horizon \
--max-model-len 32768 \
--gpu-memory-utilization 0.8 \
--override-generation-config '{"temperature": 0}'
Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.
For agent use, add --enable-auto-tool-choice --tool-call-parser k2_horizon. To serve with the sampling settings used in the evaluation below, which are also the verifier's recommended settings, swap the last flag:
--override-generation-config '{"temperature":1.0,"top_p":0.95}'
The DSpark drafter achieved 4.1x speedup on Math (109 --> 450 tokens/sec) and 3x on Code (109 --> 330 tokens/sec), even though I only used 50k samples from open-perfectblend (less than my usual 200k).
Even with sampling at temperature: 1.0 and top_p: 0.95 the speedups remain significant at 3.2x on Math and 2.4x on Code