| --- |
| library_name: sglang |
| pipeline_tag: text-generation |
| license: mit |
| base_model: |
| - inclusionAI/Ling-3.0-tiny |
| tags: |
| - speculative-decoding |
| - tree-speculative-decoding |
| - ubtree |
| - dflash |
| - sglang |
| - ling3 |
| - arxiv:2609.39972 |
| language: |
| - zh |
| - en |
| --- |
| |
| # Ling3-Tiny-UBTree-B16 |
|
|
| [Paper](https://arxiv.org/abs/2609.39972) · [Target model](https://huggingface.co/inclusionAI/Ling-3.0-tiny) |
|
|
| **Ling3-Tiny-UBTree-B16** is a UBTree speculative-decoding draft model for |
| [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny). It combines a |
| five-layer DFlash backbone with a learned bigram selector to propose multiple |
| candidate continuations for target-model verification. |
|
|
| > **SGLang integration — stay tuned!** Our UBTree adaptation for Ling3 has not |
| > yet been released to the open-source SGLang community. Public installation |
| > instructions and serving recipes will be added here when it becomes available. |
|
|
| This checkpoint is used alongside the matching target model; it cannot serve |
| as a standalone text-generation model. |
|
|
| ## How UBTree works |
|
|
|  |
|
|
| *UBTree pipeline* |
|
|
| UBTree pairs a **unigram proposer** with a **bigram selector**: |
|
|
| 1. A DFlash backbone proposes candidate tokens for future positions in parallel. |
| 2. The selector scores transitions between adjacent candidates in parallel. |
| 3. Tree construction retains promising branches within a verification budget. |
| 4. The target verifies the tree and commits the accepted continuation. |
|
|
| The paper also introduces tree-native training with renormalized KL supervision |
| on high-temperature target traces, encouraging plausible alternative branches. |
| See the [paper](https://arxiv.org/abs/2609.39972) for the method and experiments. |
|
|
| ## Model specifications |
|
|
| | Item | Value | |
| |---|---| |
| | Target model | [inclusionAI/Ling-3.0-tiny](https://huggingface.co/inclusionAI/Ling-3.0-tiny) | |
| | Draft architecture | `Ling3UBTreeModel` | |
| | Draft backbone | 5 full-attention Transformer layers | |
| | Parameters stored in this checkpoint | 183,127,326 (183.1M) | |
| | Weight dtype | BF16 | |
| | Hidden size | 1,536 | |
| | Attention heads / KV heads | 16 / 16 | |
| | Target auxiliary feature layers (zero-based) | 4, 8, 12, 16, 22 | |
| | Draft block size | 16 (anchor-inclusive runtime convention) | |
| | Candidate top-k | 64 per draft position | |
| | Default verification tree budget | 64 non-root nodes + 1 anchor | |
| | Training | DFlash warm start, then joint backbone + UBTree selector training | |
|
|
| The parameter count covers tensors distributed in this repository. The draft |
| shares token embeddings and the language-model head with the target; these |
| shared weights are not included here. B16 denotes the draft block size, not |
| the verification tree's node budget. |
|
|
| ## Serving with SGLang |
|
|
| UBTree support for Ling3 in SGLang is coming soon. We will share installation |
| instructions and serving examples once the integration is publicly available. |
| **Stay tuned!** |
|
|
| ## Benchmark results |
|
|
| All methods use **block size 16**, TP1, concurrency 1, BF16, and greedy |
| no-thinking decoding. UBTree uses a 49-node tree. |
|
|
| Each cell shows **acceptance length / E2E speedup over target-only AR**. |
|
|
| | Dataset | UBTree | DFlash | DSpark | |
| |---|---:|---:|---:| |
| | GSM8K | **9.53 / 3.055×** | 6.57 / 2.665× | 7.53 / 2.718× | |
| | MATH-500 | **9.86 / 3.518×** | 5.74 / 2.643× | 6.96 / 2.826× | |
| | AIME 2025 | **8.44 / 3.152×** | 4.76 / 2.261× | 5.37 / 2.263× | |
| | HumanEval | **10.04 / 3.411×** | 6.39 / 2.770× | 7.59 / 2.914× | |
| | MBPP | **9.35 / 3.062×** | 5.87 / 2.473× | 7.16 / 2.678× | |
| | LiveCodeBench | **7.46 / 2.794×** | 4.11 / 1.948× | 4.84 / 2.030× | |
|
|
| AL and throughput use three-round medians; the matched AR baseline uses one round. |
|
|
| ## Citation |
|
|
| If you use UBTree in your work, please cite: |
|
|
| ```bibtex |
| @article{liang2026ubtree, |
| title = {UBTree: Parallel Tree Drafting via Unigram and Bigram Models for Speculative Decoding}, |
| author = {Liang, Chumeng and Wang, Linxuan and Peng, Xinyu and Liu, Huabin and Chen, Yuxin and Liu, Ge and Lin, Guang and Song, Qifan and Li, Jianguo}, |
| journal = {arXiv preprint arXiv:2609.39972}, |
| year = {2026}, |
| url = {https://arxiv.org/abs/2609.39972} |
| } |
| ``` |
|
|
| ## License |
|
|
| The model weights are released under the **MIT License**. The pipeline figure |
| is from [Liang et al., Figure 2](https://arxiv.org/html/2609.39972v1#S2.F2), |
| reproduced without modification under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). |
|
|
