Update README.md

8e4f4b1 verified 2 days ago

6.21 kB

---
library_name: sglang
pipeline_tag: text-generation
license: mit
base_model:
  - inclusionAI/Ling-3.0-flash
tags:
  - speculative-decoding
  - tree-speculative-decoding
  - ubtree
  - dflash
  - sglang
  - ling3
  - arxiv:2609.39972
language:
  - zh
  - en
---
# Ling3-Flash-UBTree-B16

[Website](https://www.inclusion-ai.org/ubtree) · [Paper](https://www.inclusion-ai.org/ubtree/UBTree.pdf) · [Target model](https://huggingface.co/inclusionAI/Ling-3.0-flash)

**Ling3-Flash-UBTree-B16** is a UBTree speculative-decoding draft model for
[inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash). It combines a
five-layer DFlash backbone with a learned bigram selector to propose multiple
candidate continuations for target-model verification.

This checkpoint is used alongside the matching target model; it cannot serve
as a standalone text-generation model.

## How UBTree works

![UBTree architecture and tree-native training pipeline](assets/ubtree_pipeline.png)

*UBTree pipeline*

UBTree pairs a **unigram proposer** with a **bigram selector**:

1. A DFlash backbone proposes candidate tokens for future positions in parallel.
2. The selector scores transitions between adjacent candidates in parallel.
3. Tree construction retains promising branches within a verification budget.
4. The target verifies the tree and commits the accepted continuation.

The paper also introduces tree-native training with renormalized KL supervision
on high-temperature target traces, encouraging plausible alternative branches.
See the [paper](https://arxiv.org/abs/2609.39972) for the method and experiments.

## Model specifications

| Item | Value |
|---|---|
| Target model | [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) |
| Draft architecture | `Ling3UBTreeModel` |
| Draft backbone | 5 full-attention Transformer layers |
| Parameters stored in this checkpoint | 481,983,774 (482.0M) |
| Weight dtype | BF16 |
| Hidden size | 2,560 |
| Attention heads / KV heads | 32 / 32 |
| Target auxiliary feature layers (zero-based) | 1, 11, 23, 29, 35 |
| Draft block size | 16 (anchor-inclusive runtime convention) |
| Candidate top-k | 64 per draft position |
| Default verification tree budget | 64 non-root nodes + 1 anchor |
| Training | DFlash warm start, then joint backbone + UBTree selector training |

The parameter count covers tensors distributed in this repository. The draft
shares token embeddings and the language-model head with the target; these
shared weights are not included here. B16 denotes the draft block size, not
the verification tree's node budget.

## Serving with SGLang

Use the [`asahi/ubtree` SGLang branch](https://github.com/AsaHIXx/sglang/tree/asahi/ubtree) for the commands below.
Upstream integration is in progress in [sgl-project/sglang#44050](https://github.com/sgl-project/sglang/pull/44050).

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
SGLANG_OPT_FA3_VERIFY_METADATA_IN_GRAPH=1 \
SGLANG_OPT_UBTREE_FA3_TREE_ONLY_MASK=1 \
python -m sglang.launch_server \
  --model-path /path/to/ling3-flash \
  --served-model-name ling3-flash \
  --tp-size 4 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend fa3 \
  --page-size 1 \
  --context-length 4096 \
  --max-total-tokens 4096 \
  --max-running-requests 1 \
  --mem-fraction-static 0.80 \
  --max-prefill-tokens 4096 \
  --chunked-prefill-size 4096 \
  --disable-radix-cache \
  --mamba-backend triton \
  --linear-attn-backend triton \
  --linear-attn-verify-backend triton \
  --mamba-ssm-dtype float32 \
  --cuda-graph-backend-prefill disabled \
  --cuda-graph-backend-decode full \
  --cuda-graph-bs-decode 1 \
  --disable-cuda-graph-padding \
  --tool-call-parser ling3 \
  --reasoning-parser ling3 \
  --trust-remote-code \
  --speculative-algorithm UBTREE \
  --speculative-draft-model-path /path/to/ling3-flash-ubtree \
  --speculative-num-draft-tokens 32
```

## Benchmark results

All methods use **block size 16**, TP4, concurrency 1, BF16, and greedy
no-thinking decoding. UBTree uses a 32-node tree.

Each cell shows **acceptance length / E2E speedup / TPS (tok/s)**.
E2E speedup is relative to target-only autoregressive (AR) decoding.

| Dataset | UBTree | DFlash | DSpark |
|---|---:|---:|---:|
| GSM8K | **9.70 / 3.227× / 903.3** | 7.44 / 2.927× / 819.2 | 8.00 / 2.899× / 811.4 |
| MATH-500 | **9.58 / 3.761× / 1127.3** | 7.07 / 3.284× / 984.4 | 7.69 / 3.252× / 974.8 |
| AIME 2025 | **8.18 / 3.497× / 1065.6** | 5.75 / 2.864× / 872.9 | 6.39 / 2.883× / 878.8 |
| HumanEval | **10.40 / 3.479× / 1004.3** | 7.79 / 3.128× / 903.1 | 8.03 / 3.014× / 870.1 |
| MBPP | **9.55 / 3.415× / 989.5** | 7.01 / 3.011× / 872.6 | 6.89 / 2.730× / 791.1 |
| LiveCodeBench | **7.53 / 3.191× / 967.4** | 5.13 / 2.543× / 771.0 | 5.53 / 2.484× / 753.2 |
| **Avg. (6 datasets)** | **9.16 / 3.428× / 1009.6** | 6.70 / 2.960× / 870.5 | 7.09 / 2.877× / 846.6 |

TPS is total returned output tokens divided by total client HTTP time,
including prefill and generation. Acceptance length is returned output tokens
divided by target verification calls, including the anchor. Speculative AL and
TPS are independent medians over three runs; E2E speedup is speculative TPS
divided by the matched AR TPS. AR uses one run per benchmark.

Each Avg. value is the arithmetic mean of the six per-dataset values,
computed before display rounding. Average speedup is the mean of per-dataset
speedups, rather than the ratio of average TPS.

## Citation

If you use UBTree in your work, please cite:

```bibtex
@article{liang2026ubtree,
  title   = {UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding},
  author  = {Liang, Chumeng and Wang, Linxuan and Peng, Xinyu and Liu, Huabin and Chen, Yuxin and Liu, Ge and Lin, Guang and Song, Qifan and Li, Jianguo},
  journal = {arXiv preprint arXiv:2609.39972},
  year    = {2026},
  url     = {https://arxiv.org/abs/2609.39972}
}
```

## License

The model weights are released under the **MIT License**. The pipeline figure
is from [Liang et al., Figure 2](https://arxiv.org/html/2609.39972v1#S2.F2),
reproduced without modification under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).

	---
	library_name: sglang
	pipeline_tag: text-generation
	license: mit
	base_model:
	- inclusionAI/Ling-3.0-flash
	tags:
	- speculative-decoding
	- tree-speculative-decoding
	- ubtree
	- dflash
	- sglang
	- ling3
	- arxiv:2609.39972
	language:
	- zh
	- en
	---

	# Ling3-Flash-UBTree-B16

	[Website](https://www.inclusion-ai.org/ubtree) · [Paper](https://www.inclusion-ai.org/ubtree/UBTree.pdf) · [Target model](https://huggingface.co/inclusionAI/Ling-3.0-flash)

	Ling3-Flash-UBTree-B16 is a UBTree speculative-decoding draft model for
	[inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash). It combines a
	five-layer DFlash backbone with a learned bigram selector to propose multiple
	candidate continuations for target-model verification.

	This checkpoint is used alongside the matching target model; it cannot serve
	as a standalone text-generation model.

	## How UBTree works

	![UBTree architecture and tree-native training pipeline](assets/ubtree_pipeline.png)

	UBTree pipeline

	UBTree pairs a unigram proposer with a bigram selector:

	1. A DFlash backbone proposes candidate tokens for future positions in parallel.
	2. The selector scores transitions between adjacent candidates in parallel.
	3. Tree construction retains promising branches within a verification budget.
	4. The target verifies the tree and commits the accepted continuation.

	The paper also introduces tree-native training with renormalized KL supervision
	on high-temperature target traces, encouraging plausible alternative branches.
	See the [paper](https://arxiv.org/abs/2609.39972) for the method and experiments.

	## Model specifications

	\| Item \| Value \|
	\|---\|---\|
	\| Target model \| [inclusionAI/Ling-3.0-flash](https://huggingface.co/inclusionAI/Ling-3.0-flash) \|
	\| Draft architecture \| `Ling3UBTreeModel` \|
	\| Draft backbone \| 5 full-attention Transformer layers \|
	\| Parameters stored in this checkpoint \| 481,983,774 (482.0M) \|
	\| Weight dtype \| BF16 \|
	\| Hidden size \| 2,560 \|
	\| Attention heads / KV heads \| 32 / 32 \|
	\| Target auxiliary feature layers (zero-based) \| 1, 11, 23, 29, 35 \|
	\| Draft block size \| 16 (anchor-inclusive runtime convention) \|
	\| Candidate top-k \| 64 per draft position \|
	\| Default verification tree budget \| 64 non-root nodes + 1 anchor \|
	\| Training \| DFlash warm start, then joint backbone + UBTree selector training \|

	The parameter count covers tensors distributed in this repository. The draft
	shares token embeddings and the language-model head with the target; these
	shared weights are not included here. B16 denotes the draft block size, not
	the verification tree's node budget.

	## Serving with SGLang

	Use the [`asahi/ubtree` SGLang branch](https://github.com/AsaHIXx/sglang/tree/asahi/ubtree) for the commands below.
	Upstream integration is in progress in [sgl-project/sglang#44050](https://github.com/sgl-project/sglang/pull/44050).

	```bash
	CUDA_VISIBLE_DEVICES=0,1,2,3 \
	SGLANG_OPT_FA3_VERIFY_METADATA_IN_GRAPH=1 \
	SGLANG_OPT_UBTREE_FA3_TREE_ONLY_MASK=1 \
	python -m sglang.launch_server \
	--model-path /path/to/ling3-flash \
	--served-model-name ling3-flash \
	--tp-size 4 \
	--host 0.0.0.0 \
	--port 30000 \
	--attention-backend fa3 \
	--page-size 1 \
	--context-length 4096 \
	--max-total-tokens 4096 \
	--max-running-requests 1 \
	--mem-fraction-static 0.80 \
	--max-prefill-tokens 4096 \
	--chunked-prefill-size 4096 \
	--disable-radix-cache \
	--mamba-backend triton \
	--linear-attn-backend triton \
	--linear-attn-verify-backend triton \
	--mamba-ssm-dtype float32 \
	--cuda-graph-backend-prefill disabled \
	--cuda-graph-backend-decode full \
	--cuda-graph-bs-decode 1 \
	--disable-cuda-graph-padding \
	--tool-call-parser ling3 \
	--reasoning-parser ling3 \
	--trust-remote-code \
	--speculative-algorithm UBTREE \
	--speculative-draft-model-path /path/to/ling3-flash-ubtree \
	--speculative-num-draft-tokens 32
	```

	## Benchmark results

	All methods use block size 16, TP4, concurrency 1, BF16, and greedy
	no-thinking decoding. UBTree uses a 32-node tree.

	Each cell shows acceptance length / E2E speedup / TPS (tok/s).
	E2E speedup is relative to target-only autoregressive (AR) decoding.

	\| Dataset \| UBTree \| DFlash \| DSpark \|
	\|---\|---:\|---:\|---:\|
	\| GSM8K \| 9.70 / 3.227× / 903.3 \| 7.44 / 2.927× / 819.2 \| 8.00 / 2.899× / 811.4 \|
	\| MATH-500 \| 9.58 / 3.761× / 1127.3 \| 7.07 / 3.284× / 984.4 \| 7.69 / 3.252× / 974.8 \|
	\| AIME 2025 \| 8.18 / 3.497× / 1065.6 \| 5.75 / 2.864× / 872.9 \| 6.39 / 2.883× / 878.8 \|
	\| HumanEval \| 10.40 / 3.479× / 1004.3 \| 7.79 / 3.128× / 903.1 \| 8.03 / 3.014× / 870.1 \|
	\| MBPP \| 9.55 / 3.415× / 989.5 \| 7.01 / 3.011× / 872.6 \| 6.89 / 2.730× / 791.1 \|
	\| LiveCodeBench \| 7.53 / 3.191× / 967.4 \| 5.13 / 2.543× / 771.0 \| 5.53 / 2.484× / 753.2 \|
	\| Avg. (6 datasets) \| 9.16 / 3.428× / 1009.6 \| 6.70 / 2.960× / 870.5 \| 7.09 / 2.877× / 846.6 \|

	TPS is total returned output tokens divided by total client HTTP time,
	including prefill and generation. Acceptance length is returned output tokens
	divided by target verification calls, including the anchor. Speculative AL and
	TPS are independent medians over three runs; E2E speedup is speculative TPS
	divided by the matched AR TPS. AR uses one run per benchmark.

	Each Avg. value is the arithmetic mean of the six per-dataset values,
	computed before display rounding. Average speedup is the mean of per-dataset
	speedups, rather than the ratio of average TPS.

	## Citation

	If you use UBTree in your work, please cite:

	```bibtex
	@article{liang2026ubtree,
	title = {UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding},
	author = {Liang, Chumeng and Wang, Linxuan and Peng, Xinyu and Liu, Huabin and Chen, Yuxin and Liu, Ge and Lin, Guang and Song, Qifan and Li, Jianguo},
	journal = {arXiv preprint arXiv:2609.39972},
	year = {2026},
	url = {https://arxiv.org/abs/2609.39972}
	}
	```

	## License

	The model weights are released under the MIT License. The pipeline figure
	is from [Liang et al., Figure 2](https://arxiv.org/html/2609.39972v1#S2.F2),
	reproduced without modification under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).