---
license: apache-2.0
base_model: netease-youdao/Confucius3-Math
tags:
- eagle3
- speculative-decoding
- draft-model
- vllm
---

# Confucius3-Math EAGLE3 Draft Head

An [EAGLE3](https://arxiv.org/abs/2503.01840) speculative-decoding draft head for
[**netease-youdao/Confucius3-Math**](https://huggingface.co/netease-youdao/Confucius3-Math)
(Qwen2.5-14B, R1-style inline-think). Trained with [SpecForge](https://github.com/sgl-project/SpecForge)
on the same rollout data as a matched DFlash / D-PACE drafter, for a method comparison.

> ⚠️ **Intermediate checkpoint.** This is `epoch_2 / step_12000` of a planned 6-epoch
> (31,734-step) run — roughly **38%** through training. Acceptance length will still
> improve with further training. A final checkpoint will replace this when the run completes.

## Architecture

| | |
|---|---|
| architecture | `LlamaForCausalLMEagle3` (1 layer) |
| hidden size | 5120 |
| heads (q/kv) | 40 / 8 |
| intermediate | 13824 |
| draft vocab | 32000 (with `d2t`/`t2d` mapping to the 152064 target vocab) |
| aux hidden layers | `[1, 23, 44]` (SpecForge default) |
| dtype | bfloat16 |

## Training

- SpecForge EAGLE3 defaults: auto-generated draft config, lr 1e-4, warmup 0.015, grad-norm 0.5, TTT length 7.
- Copied from the matched DFlash/D-PACE run: same preformatted Confucius3-Math rollout data (148,112 rows), `confucius-r1` chat template, max-length 4096, batch-size 4, 6 epochs.

## Usage (vLLM)

```bash
vllm serve netease-youdao/Confucius3-Math \
  --speculative-config '{"method":"eagle3","model":"noctuashap/Confucius3-Math-Eagle3","num_speculative_tokens":7}' \
  --attention-backend flash_attn --trust-remote-code
```

`num_speculative_tokens` is the draft length — lower it (e.g. 2–3) for high-concurrency /
compute-bound serving, raise it (up to ~7) for low-batch / latency-oriented serving.

## Measured acceptance (gsm8k / math500, thinking mode, vLLM)

At an earlier checkpoint (step_8000) with `num_speculative_tokens=7`:

| dataset | accept length | tok/s (single stream) |
|---|---|---|
| gsm8k | ~4.3 | ~365 |
| math500 | ~4.5 | ~386 |

(Confucius3-Math is R1-style always-thinking, so thinking and non-thinking modes are equivalent.)