README.md

---
language:
- ru
- en
tags:
- speculative-decoding
- eagle3
- draft-model
base_model: t-tech/T-pro-it-2.1-FP8
---

# EAGLE-3 Draft Model for t-tech/T-pro-it-2.1-FP8

This is an EAGLE-3 draft model trained to accelerate inference of [t-tech/T-pro-it-2.1-FP8](https://huggingface.co/t-tech/T-pro-it-2.1-FP8) via speculative decoding.

**Measured speedup: ~2x** (after partial training; expected ~3-4x after full training).

## What is EAGLE-3?

EAGLE-3 ([paper](https://arxiv.org/abs/2503.01840)) is a speculative decoding method that trains a small (~1B) draft model to predict multiple tokens ahead, which are then verified by the large base model in a single forward pass. Unlike EAGLE/EAGLE-2, EAGLE-3 uses direct token prediction and multi-layer feature fusion (low/mid/high layers of the target model), enabling better scaling with training data.

## Usage

```python
import torch
from eagle3.model.ea_model import Eagle3Model

model = Eagle3Model.from_pretrained(
    base_model_path="t-tech/T-pro-it-2.1-FP8",
    eagle3_model_path="VirVen/T-pro-it-2.1-eagle3",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("t-tech/T-pro-it-2.1-FP8", trust_remote_code=True)

messages = [{"role": "user", "content": "Привет! Расскажи про квантовые компьютеры."}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
input_ids = tokenizer(text, return_tensors="pt").input_ids.cuda()

with torch.no_grad():
    output_ids = model.eagenerate(input_ids, temperature=0, max_new_tokens=512)

print(tokenizer.decode(output_ids[0], skip_special_tokens=True))
```

## Requirements

Install the EAGLE-3 package:
```bash
pip install git+https://github.com/SafeAILab/EAGLE.git
# or from your local repo:
pip install -e .
```

## Training details

- Base model: `t-tech/T-pro-it-2.1-FP8` (Qwen3-32B architecture, FP8)
- Draft model: 1B params, single transformer layer with 2×hidden_size attention input
- Feature fusion: layers 8 (low), 32 (mid), 62 (high) of the target model
- Training data: ~50k samples from saiga dataset
- Training: DeepSpeed ZeRO-2, 2× GPU, lr=5e-5