ichsanlook/qwen3-eagle

Qwen3-0.6B EAGLE-3 Draft Model

EAGLE-3 draft model trained for speculative decoding with Qwen3-0.6B as the target model.

Trained with a 60K Fortran-optimized vocabulary for 30 epochs on CPU (PyTorch), achieving:

• Train loss: 2.32
• Train accuracy: 79.7%

Files

File	Format	Size	Description
eagle3_draft_f16.gguf	GGUF F16	470 MB	Draft model in F16 precision
eagle3_draft_q8.gguf	GGUF Q8_0	238 MB	Draft model quantized to Q8_0
hf_model/	safetensors	896 MB	Hugging Face format for fine-tuning

Usage with llama.cpp

# Basic usage with speculative decoding
llama-cli \
  -m Qwen3-0.6B-target.gguf \
  -md eagle3_draft_q8.gguf \
  --spec-type draft-eagle3 \
  --spec-draft-n-max 5 \
  -p "Hello, how are you?" \
  -n 200

# Server mode (enables Ollama-like API)
llama-server \
  -m Qwen3-0.6B-target.gguf \
  -md eagle3_draft_q8.gguf \
  --spec-type draft-eagle3 \
  --spec-draft-n-max 5 \
  --port 8080

Usage with Ollama

1. Pull the target model:

ollama pull qwen3:0.6b

2. Download the draft model GGUF from this repo

3. Create a Modelfile:

FROM qwen3:0.6b
DRAFT_MODEL ./eagle3_draft_q8.gguf
PARAMETER speculative_type draft-eagle3
PARAMETER speculative_draft_max 5

4. Create and run:

ollama create qwen3-eagle -f Modelfile
ollama run qwen3-eagle

Benchmark (CPU-only, 16GB RAM)

Config	Speed (t/s)	vs Baseline
Baseline (no draft)	33.2	1.00x
Ngram (len=3)	34.6	1.04x
EAGLE-3 (len=3)	15.6	0.47x
EAGLE-3 (len=5)	11.8	0.36x

Note: EAGLE-3 speedup requires GPU acceleration. On CPU-only, the dual-model overhead outweighs speculation benefits for small models (0.6B). Speedup expected with GPU offloading or larger target models.

Training Details

• Target model: Qwen3-0.6B (1024 dim, 3072 FFN, 28 layers)
• Draft architecture: EAGLE-3 with 1 Eagle3Block (1024 hidden, 8 heads, 4 KV heads)
• Draft vocabulary: 60K tokens (Fortran-optimized selection)
• Training data: 500 ShareGPT conversations
• Loss: Cross-entropy with cosine annealing + warmup
• Hardware: CPU-only (no CUDA), PyTorch 2.13.0
• Fortran integration: Acceptance sampling, logit correction, and rejection sampling kernels

Architecture

EAGLE-3 extracts hidden states from target model layers [2, 14, 25], projects them through a feature compressor, and feeds them into a lightweight transformer block for draft prediction.