README.md

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/logos/angelslim_logo_light.png">
    <img alt="AngelSlim" src="https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/logos/angelslim_logo.png?raw=true" width=55%>
  </picture>
</p>

<h3 align="center">
A more accessible, comprehensive, and efficient toolkit for large model compression.
</h3>

<p align="center">
          ✒️ <a href="https://arxiv.org/abs/2602.21233">TechnicalReport</a>&nbsp&nbsp | &nbsp&nbsp 📖 <a href="https://angelslim.readthedocs.io/">Documentation</a>&nbsp&nbsp | &nbsp&nbsp🤗 <a href="https://huggingface.co/AngelSlim">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp🤖 <a href="https://modelscope.cn/organization/AngelSlim">ModelScope</a>
<br>
</p>

<p align="center">
          💬 <a href="./docs/source/assets/angel_slim_wechat.png">WeChat</a> | &nbsp&nbsp🫨 <a href="https://discord.com/invite/dHVNeuNdFt">Discord</a>
<br>
</p>

# Hy3-A21B DFly High-Thinking Drafter

## Model Summary

This repository contains a **DFly drafter model** trained for speculative decoding with **Hy3-A21B** under the **high-thinking** inference setting. The model is trained with **AngelSpec** and is designed to propose candidate tokens that are verified by the Hy3-A21B target model.

This drafter is not a standalone chat model. It should be deployed with the matching Hy3-A21B target model and the corresponding **high-thinking** target-side inference mode.

- **Target model:** Hy3-A21B
- **Drafter type:** DFly
- **Inference mode:** High-thinking
- **Training framework:** AngelSpec
- **Recommended serving backend:** vLLM speculative decoding
- **Recommended speculative tokens:** 8

## Training and Thinking-Mode Specialization

High-thinking and no-thinking decoding induce different hidden-state and acceptance distributions. As a result, a drafter trained for one thinking mode should not be assumed to transfer optimally to the other.

This checkpoint is the **high-thinking DFly drafter**: it is trained with high-thinking target-side data and should be evaluated and deployed with high-thinking prompts / target-side generation behavior. The matched high-thinking configuration is important for preserving acceptance length and speculative-decoding efficiency.

## Evaluation: Accepted Length

The following table reports this model's mean accepted length on Hy3-A21B under the **high-thinking** setting at **temperature 1**. The average is computed over the six displayed benchmarks.

| Target Model | Drafter | Train Data | Test Data | Math500 | GSM8K | HumanEval | MBPP | LiveCodeBench | MT-Bench | Avg. |
|---|---|---|---|---:|---:|---:|---:|---:|---:|---:|
| Hy3-A21B | DFly | High-think | High-think | 4.29 | 4.47 | 4.76 | 4.74 | 3.92 | **3.42** | 4.27 |

The high-thinking drafter is specialized for high-thinking inference. It is especially important for reasoning- and chat-oriented workloads where the target-side thinking mode changes the acceptance distribution.

## Deployment

Set `TARGET` to the Hy3-A21B target model path or repository and `DRAFT` to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.

```bash
export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-high-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dspark", "model":"$DRAFT", "num_speculative_tokens":7}"
```

## Technical Report

For more details, see the technical report:

> **AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding**  
> arXiv: https://arxiv.org/abs/2607.25852