Qwen3.8-27B-speculator.dflash2
Model Overview
- Model Architecture: Qwen3_5ForConditionalGeneration
- Input: Text / Image
- Output: Text
- Model Optimizations:
- Speculative Decoding Algorithm: DFlash2
- Draft Model: 5-layer qwen3 backbone (DFlash2DraftModel)
- Release Date: 2026-11-09
- Version: 1.0
- Model Developers: RedHatAI
This is a DFlash2 speculator model for Qwen/Qwen3.8-27B. It was trained using the Speculators library (version 0.9.0a68, commit 0d08af092bf66ca20a731a5a5338a3652dac6fa4) on a locally prepared packed chat dataset (sequence length 8192).
Training Details
Trained with the Speculators library (version 0.9.0a68, commit 0d08af0) for 5 epochs. The draft attaches to verifier layers 5, 19, 33, 47, and 61. Loss is token cross-entropy with DPACE position weighting (per_position_loss_weight=dpace). The optimizer is Muon. The DFlash2 selector uses selector_miss_policy=strict and a block size of 8. Training used 4 processes (torchrun --nproc_per_node 4) with FSDP sharding. vLLM 0.30.0 generated hidden states.
Commands
Using the Speculators library.
Prepare data
speculators prepare-data \
--model Qwen/Qwen3.8-27B \
--data <PRETOKENIZED_JSONL> \
--output <PREPARED_DATA_DIR> \
--seq-length 8192 \
--num-preprocessing-workers 64
Launch vLLM
Hidden-state server for training. Draft layers read verifier layers 5, 19, 33, 47, and 61.
python scripts/launch_vllm.py Qwen/Qwen3.8-27B \
--target-layer-ids 5 19 33 47 61 \
--port 8400 \
--hidden-states-path <HIDDEN_STATES_DIR> \
-- \
--tensor-parallel-size 2 \
--data-parallel-size 2 \
--max-model-len 20000 \
--gpu-memory-utilization 0.90 \
--trust-remote-code \
--kernel-config '{"enable_flashinfer_autotune":false}'
Launch training
Must be run once vLLM has finished launching and is running in the background.
torchrun --nproc_per_node 4 --standalone -m speculators.train \
--speculator-type dflash2 \
--scheduler-type cosine \
--selector-miss-policy strict \
--seed 42 \
--verifier-name-or-path Qwen/Qwen3.8-27B \
--num-layers 5 \
--target-layer-ids 5 19 33 47 61 \
--data-path <PREPARED_DATA_DIR> \
--total-seq-len 8192 \
--train-data-ratio 0.99 \
--max-anchors 1536 \
--num-workers 2 \
--vllm-endpoint http://localhost:8400/v1 \
--per-position-loss-weight dpace \
--on-missing generate \
--request-timeout 300 \
--max-retries 3 \
--loss-fn ce \
--optimizer muon \
--lr 6e-4 \
--muon-lr 0.003 \
--epochs 5 \
--checkpoint-freq 0.5 \
--save-path <CHECKPOINT_DIR> \
--fsdp-shard \
--logger trackio \
--log-dir <LOG_DIR> \
--run-name qwen38-dflash2_selectstrict_maxanchors1536_full_5epochs \
--block-size 8 \
--hidden-states-path <HIDDEN_STATES_DIR> \
--log-freq 400
Model Specifications
| Base Model | Qwen/Qwen3.8-27B |
| Chat Template | Qwen/Qwen3.8-27B (use /chat/completions endpoint) |
| Format | Safetensors |
| License | Apache 2.0 |
Deployment
vLLM selects the DFlash2 draft from the checkpoint architectures field. The serve method is dflash.
# Deploy with speculative decoding
vllm serve Qwen/Qwen3.8-27B \
--speculative-config '{
"model": "RedHatAI/Qwen3.8-27B-speculator.dflash2",
"num_speculative_tokens": 7,
"method": "dflash"
}'
Use the verifier chat template through the /chat/completions endpoint.
Evaluation
Per-position token acceptance rates across datasets. Positions are the draft tokens proposed at each step (acceptance_at_pos_0 through acceptance_at_pos_6). Average length is acceptance_length.
Throughput
Closed-loop output throughput versus interactivity on RedHatAI/speculator_benchmarks, HumanEval and math reasoning. The target-only and speculator curves used the same verifier, hardware, and server flags. The x-axis is 1000 / mean inter-token latency (output tokens per second per user). The y-axis is aggregate output tokens per second over the measurement window.
| Hardware | 4× NVIDIA H100 80GB, tensor parallel 4 |
| Serving | vLLM 0.30.0, method=dflash, num_speculative_tokens=7, prefix caching off, max model length 16384, max sequences 256, max batched tokens 16384, GPU memory utilization 0.92 |
| Sampling | temperature 1.0, top-p 0.95, top-k 20, taken from the verifier generation_config.json |
| Load | N = 1, 2, 4, 8, 16 with 30 s warmup and a 90 s window; N = 32, 64, 128 with 60 s warmup and a 120 s window. One run per point. max_tokens 1024 |
| Client | GuideLLM 0.7.4 through /v1/chat/completions |
Closed-loop sweep on speedbench qualitative, RAG and summarization, with the same hardware, sampling, and load schedule. Serving is vLLM 0.33.0. At one user, output throughput is 2.68× the target-only server on RAG and 2.76× on summarization. Both stay ahead at 128 concurrent requests, 1.02× and 1.03×.
Acceptance on RedHatAI/speculator_benchmarks.
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| HumanEval | 83.89% | 68.07% | 55.16% | 45.12% | 37.47% | 31.68% | 27.16% | 4.49 |
| Math Reasoning | 92.54% | 84.88% | 77.48% | 69.89% | 63.48% | 58.08% | 51.74% | 5.98 |
| QA | 80.92% | 64.68% | 51.94% | 42.02% | 34.36% | 28.16% | 22.91% | 4.25 |
| Question | 79.80% | 61.40% | 47.47% | 36.81% | 29.11% | 23.38% | 19.16% | 3.97 |
| RAG | 86.16% | 73.39% | 62.26% | 53.18% | 45.24% | 38.86% | 33.60% | 4.93 |
| Summarization | 82.91% | 66.13% | 51.20% | 39.43% | 29.85% | 22.60% | 16.34% | 4.08 |
| Tool Call | 79.28% | 59.90% | 45.12% | 34.25% | 26.34% | 20.66% | 16.23% | 3.82 |
| Translation | 85.11% | 71.65% | 59.73% | 50.15% | 41.92% | 34.82% | 28.42% | 4.72 |
| Writing | 79.56% | 60.31% | 46.20% | 35.23% | 27.14% | 21.42% | 17.05% | 3.87 |
Acceptance on speedbench qualitative.
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---|---|---|---|---|---|---|---|---|
| Coding | 80.78% | 62.86% | 48.88% | 38.50% | 30.81% | 25.12% | 20.81% | 4.08 |
| Humanities | 80.23% | 64.00% | 50.90% | 40.88% | 33.13% | 27.29% | 22.55% | 4.19 |
| Math | 77.19% | 58.93% | 44.63% | 34.31% | 26.47% | 20.64% | 16.04% | 3.78 |
| Multilingual | 79.96% | 64.46% | 52.37% | 42.91% | 35.61% | 29.84% | 25.19% | 4.30 |
| QA | 81.87% | 66.38% | 54.52% | 45.29% | 38.28% | 32.71% | 28.23% | 4.47 |
| RAG | 85.26% | 71.05% | 58.64% | 49.55% | 41.78% | 35.60% | 30.64% | 4.73 |
| Reasoning | 81.71% | 64.14% | 50.21% | 39.79% | 31.42% | 25.53% | 20.80% | 4.14 |
| Roleplay | 77.91% | 58.38% | 44.19% | 34.27% | 27.74% | 23.03% | 19.64% | 3.85 |
| STEM | 79.63% | 61.14% | 46.70% | 36.01% | 28.14% | 22.19% | 17.74% | 3.92 |
| Summarization | 85.13% | 69.91% | 56.80% | 45.81% | 37.33% | 30.89% | 25.21% | 4.51 |
| Writing | 76.75% | 56.74% | 42.59% | 32.44% | 25.56% | 20.42% | 16.75% | 3.71 |
References
DFlash2 extends block drafting from DFlash with local dynamic convolutions and a candidate selector.
Paper: DFlash: Block Diffusion for Flash Speculative Decoding
- Downloads last month
- 119
Model tree for RedHatAI/Qwen3.8-27B-speculator.dflash2
Base model
Qwen/Qwen3.8-27B
