README.md

<p align="center">
  <picture>
    <source media="(prefers-color-scheme: dark)" srcset="./docs/source/assets/logos/angelslim_logo_light.png">
    <img alt="AngelSlim" src="https://github.com/Tencent/AngelSlim/blob/main/docs/source/assets/logos/angelslim_logo.png?raw=true" width=55%>
  </picture>
</p>

<h3 align="center">
A more accessible, comprehensive, and efficient toolkit for large model compression.
</h3>

<p align="center">
          ✒️ <a href="https://arxiv.org/abs/2602.21233">TechnicalReport</a>&nbsp&nbsp | &nbsp&nbsp 📖 <a href="https://angelslim.readthedocs.io/">Documentation</a>&nbsp&nbsp | &nbsp&nbsp🤗 <a href="https://huggingface.co/AngelSlim">Hugging Face</a>&nbsp&nbsp | &nbsp&nbsp🤖 <a href="https://modelscope.cn/organization/AngelSlim">ModelScope</a>
<br>
</p>

<p align="center">
          💬 <a href="./docs/source/assets/angel_slim_wechat.png">WeChat</a> | &nbsp&nbsp🫨 <a href="https://discord.com/invite/dHVNeuNdFt">Discord</a>
<br>
</p>

# Hy3-A21B No-Thinking Drafters: MTP and DFly

## Model Summary

This repository contains no-thinking speculative-decoding drafter checkpoints for **Hy3-A21B**, including an **MTP drafter** and a **DFly drafter**. These draft models are trained with **AngelSpec** and are intended to accelerate Hy3-A21B inference by proposing candidate tokens that are verified by the target model.

They should be deployed together with the corresponding Hy3-A21B target model under the **no-thinking** inference setting.

- **Target model:** Hy3-A21B
- **Inference mode:** No-thinking
- **Drafter types:** MTP and DFly
- **Training / inference framework:** AngelSpec
- **Recommended serving backend:** vLLM speculative decoding
- **Recommended MTP speculative tokens:** 3
- **Recommended DFly speculative tokens:** 8

## Training Data

The no-thinking Hy3 drafter training starts from Open-PerfectBlend-style instruction prompts. Target responses are regenerated with **Hy3-A21B** under the no-thinking setting so that the drafter learns the target-side token distribution used during deployment.

### Domain-Specific Data Expansion for DFly

To further strengthen code generation and mathematical reasoning, the DFly training corpus augments Open-PerfectBlend with **700K domain-specific prompts**:

- **500K code prompts** from OpenCodeInstruct and OpenCodeReasoning.
  - OpenCodeInstruct provides broad code-instruction coverage.
  - OpenCodeReasoning emphasizes distilled reasoning traces for competitive programming.
  - Together, they improve both general code completion and multi-step program synthesis.
- **200K math prompts** from Big-Math, adding coverage of diverse and challenging reasoning problems.

Before response generation, prompts that share a contiguous sequence of at least **16 tokens** with an evaluation example are removed. Responses for the filtered prompts are then regenerated with **Hy3-A21B**, mixed with Open-PerfectBlend, and used to train the combined corpus for six epochs with the same objective.

This expansion mainly improves math/code acceptance length. The gain is consistent across structured benchmarks, while chat-oriented performance may not improve uniformly because the final checkpoint is specialized toward code and math.

## Evaluation: Accepted Length

The following table reports the mean accepted length on Hy3-A21B at **temperature 1** under the **no-thinking** setting. The average is the arithmetic mean over the six benchmarks.

| Target Model | Drafter | Math500 | GSM8K | HumanEval | MBPP | LiveCodeBench | MT-Bench | Avg. |
|---|---|---:|---:|---:|---:|---:|---:|---:|
| Hy3-A21B | MTP | 3.30 | 3.30 | 3.13 | 3.04 | 2.84 | 2.40 | 3.00 |
| Hy3-A21B | DFlash | 4.01 | 4.23 | 4.36 | 4.05 | 3.10 | 2.38 | 3.69 |
| Hy3-A21B | DFly | **5.23** | **5.53** | **5.52** | **5.41** | **4.07** | **2.96** | **4.79** |

## Evaluation: Throughput Benefit

The following results are measured on **HY3 295B-A21B with TP=8** at **temperature 1** across concurrency levels. `Tok/s` denotes output-token throughput. `Spd.` denotes throughput relative to autoregressive decoding (`AR`). Each cell uses `3 x 120s` measurement windows, and `Avg.` is the arithmetic mean across the six datasets.

| Conc. | Method | GSM8K Tok/s | GSM8K Spd. | Math500 Tok/s | Math500 Spd. | HumanEval Tok/s | HumanEval Spd. | MBPP Tok/s | MBPP Spd. | LiveCodeBench Tok/s | LiveCodeBench Spd. | MT-Bench Tok/s | MT-Bench Spd. | Avg. Tok/s | Avg. Spd. |
|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| c4 | AR | 287.9 | 1.00x | 293.9 | 1.00x | 279.7 | 1.00x | 294.4 | 1.00x | 284.5 | 1.00x | 290.0 | 1.00x | 288.4 | 1.00x |
| c4 | MTP-3 | 495.5 | 1.72x | 517.0 | 1.76x | 473.6 | 1.69x | 476.7 | 1.62x | 414.0 | 1.46x | 384.5 | **1.33x** | 460.2 | 1.60x |
| c4 | DFlash-8 | 569.5 | 1.98x | 587.9 | 2.00x | 588.1 | 2.10x | 575.7 | 1.96x | 408.4 | 1.44x | 371.6 | 1.28x | 516.9 | 1.79x |
| c4 | DFly-8 | 635.4 | **2.21x** | 643.9 | **2.19x** | 647.2 | **2.31x** | 661.5 | **2.25x** | 455.2 | **1.60x** | 384.0 | 1.32x | 571.2 | **1.98x** |
| c8 | AR | 426.3 | 1.00x | 435.7 | 1.00x | 400.3 | 1.00x | 433.7 | 1.00x | 413.4 | 1.00x | 421.8 | 1.00x | 421.8 | 1.00x |
| c8 | MTP-3 | 756.9 | 1.78x | 791.5 | 1.82x | 717.4 | 1.79x | 729.1 | 1.68x | 620.8 | 1.50x | 590.1 | **1.40x** | 701.0 | 1.66x |
| c8 | DFlash-8 | 857.0 | 2.01x | 860.1 | 1.97x | 866.2 | 2.16x | 866.2 | 2.00x | 609.1 | 1.47x | 545.5 | 1.29x | 767.4 | 1.82x |
| c8 | DFly-8 | 964.8 | **2.26x** | 959.5 | **2.20x** | 974.1 | **2.43x** | 1001.5 | **2.31x** | 641.3 | **1.55x** | 565.8 | 1.34x | 851.2 | **2.02x** |
| c16 | AR | 650.7 | 1.00x | 670.6 | 1.00x | 594.8 | 1.00x | 667.9 | 1.00x | 617.2 | 1.00x | 628.1 | 1.00x | 638.2 | 1.00x |
| c16 | MTP-3 | 1146.6 | 1.76x | 1174.0 | 1.75x | 1076.1 | 1.81x | 1103.3 | 1.65x | 893.5 | 1.45x | 876.6 | 1.40x | 1045.0 | 1.64x |
| c16 | DFlash-8 | 1446.1 | 2.22x | 1430.4 | 2.13x | 1433.3 | 2.41x | 1440.4 | 2.16x | 970.5 | 1.57x | 901.0 | 1.43x | 1270.3 | 1.99x |
| c16 | DFly-8 | 1623.8 | **2.50x** | 1607.5 | **2.40x** | 1595.4 | **2.68x** | 1677.8 | **2.51x** | 1046.7 | **1.70x** | 961.5 | **1.53x** | 1418.8 | **2.22x** |
| c32 | AR | 918.6 | 1.00x | 1000.8 | 1.00x | 850.8 | 1.00x | 1018.7 | 1.00x | 893.2 | 1.00x | 921.1 | 1.00x | 933.9 | 1.00x |
| c32 | MTP-3 | 1923.8 | 2.09x | 1989.8 | 1.99x | 1780.7 | 2.09x | 1863.6 | 1.83x | 1394.2 | 1.56x | 1469.4 | 1.60x | 1736.9 | 1.86x |
| c32 | DFlash-8 | 2261.8 | 2.46x | 2334.3 | 2.33x | 2170.6 | 2.55x | 2339.3 | 2.30x | 1489.9 | 1.67x | 1453.4 | 1.58x | 2008.2 | 2.15x |
| c32 | DFly-8 | 2527.4 | **2.75x** | 2608.9 | **2.61x** | 2429.1 | **2.86x** | 2741.3 | **2.69x** | 1602.6 | **1.79x** | 1549.4 | **1.68x** | 2243.1 | **2.40x** |
| c64 | AR | 1156.9 | 1.00x | 1381.3 | 1.00x | 1197.6 | 1.00x | 1513.9 | 1.00x | 1229.6 | 1.00x | 1306.9 | 1.00x | 1297.7 | 1.00x |
| c64 | MTP-3 | 2932.0 | **2.53x** | 3170.0 | 2.29x | 2639.6 | 2.20x | 3011.3 | 1.99x | 2050.4 | 1.67x | 2350.0 | **1.80x** | 2692.2 | 2.08x |
| c64 | DFlash-8 | 2523.4 | 2.18x | 2947.1 | 2.13x | 2655.2 | 2.22x | 2671.8 | 1.76x | 2015.8 | 1.64x | 1815.8 | 1.39x | 2438.2 | 1.89x |
| c64 | DFly-8 | 2827.8 | 2.44x | 3301.8 | **2.39x** | 2965.0 | **2.48x** | 3130.9 | **2.07x** | 2195.0 | **1.79x** | 1936.6 | 1.48x | 2726.2 | **2.11x** |

## Deployment

Set `TARGET` to the Hy3-A21B target model path or repository and `DRAFT` to the corresponding drafter checkpoint. The relevant inference code can be found in PR https://github.com/vllm-project/vllm/pull/50246.

### MTP-3

```bash
export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-mtp-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"mtp", "model":"$DRAFT", "num_speculative_tokens":3}"
```

### DFlash-8

```bash
export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-no-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dflash", "model":"$DRAFT","num_speculative_tokens":7}"
```

### DFly-8

```bash
export TARGET="<path-or-hf-repo-to-hy3-a21b-target>"
export DRAFT="<path-or-hf-repo-to-hy3-dfly-no-thinking-drafter>"

vllm serve "$TARGET" \
  --tensor-parallel-size 8 \
  --speculative-config "{"method":"dspark", "model":"$DRAFT","num_speculative_tokens":7}"
```

## Technical Report

For more details, see the technical report:

> **AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding**  
> arXiv: https://arxiv.org/abs/2607.25852