RedHatAI/Qwen3-4B-speculator.dspark

This is a DSpark speculator model for Qwen/Qwen3-4B.

Training Details

This model was trained using the Speculators library on mlabonne/open-perfectblend. Responses were regenerated by Qwen3-8B. DSpark extends DFlash with a Markov head (intra-block token dependency) and a confidence head (per-position acceptance prediction).

Commands

Using the Speculators library and the helper scripts provided in the repo.

Prepare data

# In virtual environment with speculators installed
python scripts/prepare_data.py \
  --model Qwen/Qwen3-4B \
  --data ./data/qwen3_4b_regen/magpie_output.jsonl \
  --data ./data/qwen3_4b_regen/ultrachat_output.jsonl \
  --output ./output/dspark_qwen3_4b \
  --seq-length 4096

Launch vLLM

# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0 python scripts/launch_vllm.py \
  Qwen/Qwen3-4B \
  --target-layer-ids 1 9 17 25 33 \
  -- --port 8000 \
  --max-model-len 4098 \
  --enforce-eager

Launch training

Must be run once vLLM has finished launching and is running in the background.

# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=1,2,3,4,5,6,7 torchrun \
  --standalone \
  --nproc_per_node 7 \
  scripts/train.py \
  --verifier-name-or-path Qwen/Qwen3-4B \
  --speculator-type dspark \
  --data-path ./output/dspark_qwen3_4b \
  --vllm-endpoint http://localhost:8000/v1 \
  --save-path ./output/dspark_qwen3_4b/checkpoints \
  --block-size 7 \
  --max-anchors 512 \
  --target-layer-ids 1 9 17 25 33 \
  --num-layers 5 \
  --markov-rank 256 \
  --markov-head-type vanilla \
  --enable-confidence-head \
  --confidence-head-with-markov \
  --loss-fn '{"ce": 0.1, "tv": 0.9}' \
  --confidence-head-alpha 1.0 \
  --dflash-decay-gamma 4.0 \
  --optimizer adamw \
  --lr 6e-4 \
  --weight-decay 0.0 \
  --scheduler-type cosine \
  --scheduler-warmup-ratio 0.04 \
  --epochs 10 \
  --total-seq-len 4096 \
  --seed 42 \
  --fsdp-shard \
  --on-missing generate \
  --on-generate delete \
  --checkpoint-freq 0.1

Model Specifications

Base Model	Qwen/Qwen3-4B
Chat Template	Qwen/Qwen3-4B (use /chat/completions endpoint)
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H100

Deployment

# Deploy with speculative decoding
vllm serve Qwen/Qwen3-4B \
    --speculative-config '{
        "model": "RedHatAI/Qwen3-4B-speculator.dspark",
        "num_speculative_tokens": 8,
        "method": "dspark"
    }'

Acceptance Rates

Per-position token acceptance rates across datasets:

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Pos 7	Avg. Length
HumanEval	85.6%	71.5%	58.9%	48.2%	39.1%	31.4%	25.0%	19.6%	4.79
math_reasoning	87.7%	75.7%	65.1%	56.3%	48.2%	40.9%	34.6%	28.8%	5.37
qa	75.9%	56.7%	41.7%	31.4%	23.6%	17.6%	13.0%	9.6%	3.70
question	79.0%	60.6%	46.6%	35.9%	27.9%	21.7%	17.2%	13.2%	4.02
rag	78.0%	59.8%	45.0%	33.7%	25.0%	18.4%	13.1%	9.5%	3.82
summarization	73.7%	51.7%	35.2%	23.4%	15.4%	9.7%	6.2%	4.0%	3.19
tool_call	78.9%	60.3%	45.8%	34.3%	25.7%	19.0%	14.2%	10.3%	3.89
translation	83.4%	68.2%	54.5%	42.2%	33.4%	25.7%	19.9%	15.0%	4.42
writing	79.0%	60.7%	46.8%	36.1%	28.0%	21.9%	17.2%	13.2%	4.03