RedHatAI/Mellum2-12B-A2.5B-Thinking-Dflash

This is a DFlash speculator model for JetBrains/Mellum2-12B-A2.5B-Thinking with sliding window attention.

It was trained using the Speculators library on a combination of the Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered dataset and the train_sft split of the HuggingFaceH4/ultrachat_200k dataset. Responses were regenerated by JetBrains/Mellum2-12B-A2.5B-Thinking.

Model Specifications


Base Model	JetBrains/Mellum2-12B-A2.5B-Thinking
Format	Safetensors
License	Apache 2.0
Validation Hardware	Nvidia H100

Training Details

This model was trained using the Speculators library and the helper scripts provided in the repo.

Launch vLLM

python3 launch_vllm.py JetBrains/Mellum2-12B-A2.5B-Thinking \
  --hidden-states-path /tmp \
  --target-layer-ids 1 6 12 17 23 28 \
  -- \
  --max-model-len 16384 \
  -dp 2 \
  --max_num_batched_tokens 16384 \
  --max_num_seqs 4096 \
  --gpu-memory-utilization 0.9 \
  --no-enable-chunked-prefill

Launch Training

Must be run once vLLM has finished launching and is running in the background.

torchrun \
  --standalone \
  --nproc_per_node=2 scripts/train.py \
  --verifier-name-or-path JetBrains/Mellum2-12B-A2.5B-Thinking \
  --data-path "prepared_data" \
  --on-missing generate \
  --on-generate delete \
  --scheduler-type cosine \
  --draft-vocab-size 32000 \
  --max-anchors 3072 \
  --target-layer-ids 1 6 12 17 23 \
  --speculator-type dflash \
  --num-layers 5 \
  --logger trackio \
  --run-name demo_qwen3_9b \
  --lr 0.0006 \
  --epochs 10 \
  --num-workers 16 \
  --sliding-window-indices 0 1 2 3 4

Deployment

Deploy with vLLM using the speculator as a draft model.

First install vLLM nightly:

uv pip install -U vllm \
    --torch-backend=auto \
    --extra-index-url https://wheels.vllm.ai/nightly

Then run:

vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
    --speculative-config '{ \
        "model": "RedHatAI/Mellum2-12B-A2.5B-Thinking-Dflash", \
        "num_speculative_tokens": 16, \
        "method": "dflash" \
    }'

Preliminary Evaluations

Per-Position Acceptance Rate

Dataset	Pos 0	Pos 1	Pos 2	Pos 3	Pos 4	Pos 5	Pos 6	Avg. Length
HumanEval	72.5%	50.9%	34.3%	23.4%	16.4%	11.4%	7.6%	3.17
math_reasoning	80.9%	63.3%	48.5%	36.8%	26.7%	19.1%	12.9%	3.88
qa	64.1%	38.4%	23.0%	13.7%	8.3%	5.0%	2.4%	2.55
question	64.9%	40.1%	24.3%	15.1%	9.3%	5.8%	3.4%	2.63
rag	64.9%	39.4%	24.0%	13.6%	7.8%	4.3%	2.3%	2.56
summarization	58.5%	31.9%	17.3%	9.3%	5.2%	3.0%	1.7%	2.27
tool_call	68.8%	43.1%	26.5%	15.5%	9.1%	5.4%	3.1%	2.71
translation	58.4%	33.6%	18.9%	10.0%	5.5%	2.9%	1.5%	2.31
writing	66.2%	40.9%	25.3%	15.8%	10.2%	6.3%	3.8%	2.68