Qwen3.8-27B-speculator.dflash2

Model Overview

  • Model Architecture: Qwen3_5ForConditionalGeneration
    • Input: Text / Image
    • Output: Text
  • Model Optimizations:
    • Speculative Decoding Algorithm: DFlash2
    • Draft Model: 5-layer qwen3 backbone (DFlash2DraftModel)
  • Release Date: 2026-11-09
  • Version: 1.0
  • Model Developers: RedHatAI

This is a DFlash2 speculator model for Qwen/Qwen3.8-27B. It was trained using the Speculators library (version 0.9.0a68, commit 0d08af092bf66ca20a731a5a5338a3652dac6fa4) on a locally prepared packed chat dataset (sequence length 8192).

Training Details

Trained with the Speculators library (version 0.9.0a68, commit 0d08af0) for 5 epochs. The draft attaches to verifier layers 5, 19, 33, 47, and 61. Loss is token cross-entropy with DPACE position weighting (per_position_loss_weight=dpace). The optimizer is Muon. The DFlash2 selector uses selector_miss_policy=strict and a block size of 8. Training used 4 processes (torchrun --nproc_per_node 4) with FSDP sharding. vLLM 0.30.0 generated hidden states.

Commands

Using the Speculators library.

Prepare data

speculators prepare-data \
  --model Qwen/Qwen3.8-27B \
  --data <PRETOKENIZED_JSONL> \
  --output <PREPARED_DATA_DIR> \
  --seq-length 8192 \
  --num-preprocessing-workers 64

Launch vLLM

Hidden-state server for training. Draft layers read verifier layers 5, 19, 33, 47, and 61.

python scripts/launch_vllm.py Qwen/Qwen3.8-27B \
  --target-layer-ids 5 19 33 47 61 \
  --port 8400 \
  --hidden-states-path <HIDDEN_STATES_DIR> \
  -- \
  --tensor-parallel-size 2 \
  --data-parallel-size 2 \
  --max-model-len 20000 \
  --gpu-memory-utilization 0.90 \
  --trust-remote-code \
  --kernel-config '{"enable_flashinfer_autotune":false}'

Launch training

Must be run once vLLM has finished launching and is running in the background.

torchrun --nproc_per_node 4 --standalone -m speculators.train \
  --speculator-type dflash2 \
  --scheduler-type cosine \
  --selector-miss-policy strict \
  --seed 42 \
  --verifier-name-or-path Qwen/Qwen3.8-27B \
  --num-layers 5 \
  --target-layer-ids 5 19 33 47 61 \
  --data-path <PREPARED_DATA_DIR> \
  --total-seq-len 8192 \
  --train-data-ratio 0.99 \
  --max-anchors 1536 \
  --num-workers 2 \
  --vllm-endpoint http://localhost:8400/v1 \
  --per-position-loss-weight dpace \
  --on-missing generate \
  --request-timeout 300 \
  --max-retries 3 \
  --loss-fn ce \
  --optimizer muon \
  --lr 6e-4 \
  --muon-lr 0.003 \
  --epochs 5 \
  --checkpoint-freq 0.5 \
  --save-path <CHECKPOINT_DIR> \
  --fsdp-shard \
  --logger trackio \
  --log-dir <LOG_DIR> \
  --run-name qwen38-dflash2_selectstrict_maxanchors1536_full_5epochs \
  --block-size 8 \
  --hidden-states-path <HIDDEN_STATES_DIR> \
  --log-freq 400

Model Specifications

Base Model Qwen/Qwen3.8-27B
Chat Template Qwen/Qwen3.8-27B (use /chat/completions endpoint)
Format Safetensors
License Apache 2.0

Deployment

vLLM selects the DFlash2 draft from the checkpoint architectures field. The serve method is dflash.

# Deploy with speculative decoding
vllm serve Qwen/Qwen3.8-27B \
    --speculative-config '{
        "model": "RedHatAI/Qwen3.8-27B-speculator.dflash2",
        "num_speculative_tokens": 7,
        "method": "dflash"
    }'

Use the verifier chat template through the /chat/completions endpoint.

Evaluation

Per-position token acceptance rates across datasets. Positions are the draft tokens proposed at each step (acceptance_at_pos_0 through acceptance_at_pos_6). Average length is acceptance_length.

Throughput

Closed-loop output throughput versus interactivity on RedHatAI/speculator_benchmarks, HumanEval and math reasoning. The target-only and speculator curves used the same verifier, hardware, and server flags. The x-axis is 1000 / mean inter-token latency (output tokens per second per user). The y-axis is aggregate output tokens per second over the measurement window.

Output throughput vs. interactivity on HumanEval and math reasoning, 4×H100, TP4

Hardware 4× NVIDIA H100 80GB, tensor parallel 4
Serving vLLM 0.30.0, method=dflash, num_speculative_tokens=7, prefix caching off, max model length 16384, max sequences 256, max batched tokens 16384, GPU memory utilization 0.92
Sampling temperature 1.0, top-p 0.95, top-k 20, taken from the verifier generation_config.json
Load N = 1, 2, 4, 8, 16 with 30 s warmup and a 90 s window; N = 32, 64, 128 with 60 s warmup and a 120 s window. One run per point. max_tokens 1024
Client GuideLLM 0.7.4 through /v1/chat/completions

Closed-loop sweep on speedbench qualitative, RAG and summarization, with the same hardware, sampling, and load schedule. Serving is vLLM 0.33.0. At one user, output throughput is 2.68× the target-only server on RAG and 2.76× on summarization. Both stay ahead at 128 concurrent requests, 1.02× and 1.03×.

Output throughput vs. interactivity on speedbench RAG and summarization, 4×H100, TP4, vLLM 0.33

Acceptance on RedHatAI/speculator_benchmarks.

Dataset Pos 0 Pos 1 Pos 2 Pos 3 Pos 4 Pos 5 Pos 6 Avg. Length
HumanEval 83.89% 68.07% 55.16% 45.12% 37.47% 31.68% 27.16% 4.49
Math Reasoning 92.54% 84.88% 77.48% 69.89% 63.48% 58.08% 51.74% 5.98
QA 80.92% 64.68% 51.94% 42.02% 34.36% 28.16% 22.91% 4.25
Question 79.80% 61.40% 47.47% 36.81% 29.11% 23.38% 19.16% 3.97
RAG 86.16% 73.39% 62.26% 53.18% 45.24% 38.86% 33.60% 4.93
Summarization 82.91% 66.13% 51.20% 39.43% 29.85% 22.60% 16.34% 4.08
Tool Call 79.28% 59.90% 45.12% 34.25% 26.34% 20.66% 16.23% 3.82
Translation 85.11% 71.65% 59.73% 50.15% 41.92% 34.82% 28.42% 4.72
Writing 79.56% 60.31% 46.20% 35.23% 27.14% 21.42% 17.05% 3.87

Acceptance on speedbench qualitative.

Dataset Pos 0 Pos 1 Pos 2 Pos 3 Pos 4 Pos 5 Pos 6 Avg. Length
Coding 80.78% 62.86% 48.88% 38.50% 30.81% 25.12% 20.81% 4.08
Humanities 80.23% 64.00% 50.90% 40.88% 33.13% 27.29% 22.55% 4.19
Math 77.19% 58.93% 44.63% 34.31% 26.47% 20.64% 16.04% 3.78
Multilingual 79.96% 64.46% 52.37% 42.91% 35.61% 29.84% 25.19% 4.30
QA 81.87% 66.38% 54.52% 45.29% 38.28% 32.71% 28.23% 4.47
RAG 85.26% 71.05% 58.64% 49.55% 41.78% 35.60% 30.64% 4.73
Reasoning 81.71% 64.14% 50.21% 39.79% 31.42% 25.53% 20.80% 4.14
Roleplay 77.91% 58.38% 44.19% 34.27% 27.74% 23.03% 19.64% 3.85
STEM 79.63% 61.14% 46.70% 36.01% 28.14% 22.19% 17.74% 3.92
Summarization 85.13% 69.91% 56.80% 45.81% 37.33% 30.89% 25.21% 4.51
Writing 76.75% 56.74% 42.59% 32.44% 25.56% 20.42% 16.75% 3.71

References

DFlash2 extends block drafting from DFlash with local dynamic convolutions and a candidate selector.

Paper: DFlash: Block Diffusion for Flash Speculative Decoding

Downloads last month
119
Safetensors
Model size
2B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for RedHatAI/Qwen3.8-27B-speculator.dflash2

Base model

Qwen/Qwen3.8-27B
Finetuned
(958)
this model

Paper for RedHatAI/Qwen3.8-27B-speculator.dflash2