Qwen3.8-27B-Quark-DFlash2

A DFlash2 speculative-decoding draft model for Qwen3.8-27B, trained with AMD Quark.

  • Target model: Qwen3.8-27B (hybrid; served with --language-model-only)
  • Training data: the whole Tulu-3 SFT mixture, thinking disabled, shuffled, 3 epochs (17,118 steps)
  • This checkpoint: end of epoch 3
  • Draft architecture: 5 non-causal layers, hidden 5120, intermediate 8192, 40 heads / 8 KV heads, head dim 128, block size 8 (7 speculative tokens), grouped conv (kernel 2, group 16), candidate selector (rank 256, top-k 16)
  • Target layers tapped: 3, 15, 31, 47, 59. Mask token id: 248077
  • Parameters: about 1.27B, bfloat16

Usage with vLLM

Serve with the speculative method dflash and num_speculative_tokens = 7:

vllm serve Qwen/Qwen3.8-27B --language-model-only \
  --speculative-config '{"method": "dflash", "model": "ZachWangAMD/Qwen3.8-27B-Quark-DFlash2", "num_speculative_tokens": 7}'

This needs a vLLM build with DFlash2 support (DFlash2DraftModel).

Spec-Bench (K=7)

Metric Value
Acceptance length (AL) 3.67
Speedup vs. target alone 2.31x

Per-position acceptance: position 0 = 0.781, position 1 = 0.568, position 2 = 0.415, position 3 = 0.312.

The conv layers are part of the trained weights. Setting conv_kernel_size to 1 at inference breaks the draft (AL falls to 1.79), so do not change it.

Downloads last month
92
Safetensors
Model size
1B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for amd/Qwen3.8-27B-Quark-DFlash2

Base model

Qwen/Qwen3.8-27B
Finetuned
(956)
this model