RedHatAI/Qwen3-30B-A3B-speculator.dflash

This is a DFlash speculator model for Qwen/Qwen3-30B-A3B.

Training Details

This model was trained using the Speculators library on a subset of Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered and the train_sft split of HuggingFaceH4/ultrachat_200k.Responses were regenerated by Qwen/Qwen3-235B-A22B and stored at Qwen3_235B_base.

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-30B-A3B
  --data ./regenerated_data.jsonl \
  --output ./output \
  --assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
  --seq-length 16384

Launch vLLM

# In (separate) virutal environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
  Qwen/Qwen3-30B-A3B \
  --target-layer-ids 1 12 23 34 45 \
  --max-model-len  32768 \
  --max-num-batched-tokens 32768\
  --tensor-parallel-size 2 \
  --no-enable-chunked-prefill

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=2,3 torchrun \
  --standalone \
  --nproc_per_node 2 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-30B-A3B \     
  --data-path ./output \    
  --on-missing generate \    
  --on-generate delete \    
  --scheduler-type cosine \    
  --draft-vocab-size 32000 \    
  --max-anchors 1024 \    
  --target-layer-ids 1 12 23 34 45 \
  --speculator-type dflash \    
  --num-layers 5 \    
  --logger trackio  \    
  --lr 0.0006 \    
  --epochs 5 \    
  --sliding-window 2048 \    
  --sliding-window-indices 0 1 2 3 4 \    
  --draft-hidden-act silu 

Model Specifications

Base Model	Qwen/Qwen3-30B-A3B
Chat Template	Qwen/Qwen3-30B-A3B (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia A100

Deployment

# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git     
                                                                                                                                                                                                                                                                                                          
# Deploy with speculative decoding                                                                                                                                                                                                                                                                        
vllm serve Qwen/Qwen3-30B-A3B \                                                                                                                                                                                                                                                                                
    --tensor-parallel-size 2 \                                                                                                                                                                                                                                                                            
    --max-num-batched-tokens 32768 \
    --attention-backend FLASH_ATTN \ 
    --speculative-config '{                                                                                                                                                                                                                                                                               
        "model": "RedHatAI/Qwen3-30B-A3B-speculator.dflash",                                                                                                                                                                                                                                                   
        "num_speculative_tokens": 15,                                                                                                                                                                                                                                                                      
        "method": "dflash"                                                                                                                                                                                                                                                                                
    }'

Preliminary Evaluations

Per-position token acceptance rates across datasets:
(with reasoning enabled)

Dataset	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Pos 8	Pos 9	Pos 10	Pos 11	Pos 12	Pos 13	Pos 14	Pos 15	Avg Acceptance Rate
HumanEval	75.89%	50.00%	28.33%	17.07%	10.07%	5.91%	3.59%	2.35%	1.48%	1.23%	1.10%	1.05%	1.04%	1.02%	1.01%	13.40%
math_reasoning	81.55%	58.93%	41.26%	27.69%	18.14%	11.01%	6.08%	3.07%	1.41%	0.58%	0.18%	0.06%	0.01%	0.00%	0.00%	16.64%
qa	67.88%	39.38%	20.96%	10.35%	4.87%	2.20%	0.87%	0.38%	0.11%	0.04%	0.01%	0.00%	0.00%	0.00%	0.00%	9.81%
question	73.53%	46.00%	26.97%	15.37%	8.62%	4.62%	2.30%	1.05%	0.41%	0.13%	0.03%	0.00%	0.00%	0.00%	0.00%	11.93%
rag	72.66%	45.06%	25.47%	13.48%	6.83%	3.32%	1.39%	0.59%	0.19%	0.04%	0.01%	0.00%	0.00%	0.00%	0.00%	11.30%
summarization	66.98%	36.52%	17.98%	8.25%	3.51%	1.37%	0.49%	0.12%	0.02%	0.00%	0.00%	0.00%	0.00%	0.00%	0.00%	9.03%
tool_call	71.85%	44.02%	24.62%	13.09%	6.65%	3.49%	1.74%	0.73%	0.29%	0.14%	0.01%	0.00%	0.00%	0.00%	0.00%	11.12%
translation	67.64%	40.76%	21.68%	9.99%	4.66%	2.11%	0.90%	0.39%	0.11%	0.02%	0.00%	0.00%	0.00%	0.00%	0.00%	9.86%
writing	73.99%	46.44%	27.41%	15.78%	8.90%	4.72%	2.36%	1.13%	0.45%	0.19%	0.07%	0.01%	0.00%	0.00%	0.00%	12.09%

References

Paper: DFlash: Block Diffusion for Flash Speculative Decoding