|
|
--- |
|
|
datasets: |
|
|
- d3LLM/trajectory_data_llada_32 |
|
|
pipeline_tag: text-generation |
|
|
tags: |
|
|
- diffusion |
|
|
- text-generation |
|
|
- fast-inference |
|
|
- d3llm |
|
|
license: apache-2.0 |
|
|
library_name: transformers |
|
|
base_model: GSAI-ML/LLaDA-8B-Instruct |
|
|
--- |
|
|
|
|
|
# d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation 🚀 |
|
|
|
|
|
This repository contains **d3LLM-LLaDA**, an ultra-fast diffusion language model presented in the paper [d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation](https://huggingface.co/papers/2601.07568). |
|
|
|
|
|
- 📄 **Paper:** [arXiv:2601.07568](https://huggingface.co/papers/2601.07568) |
|
|
- 💻 **Code:** [GitHub - hao-ai-lab/d3LLM](https://github.com/hao-ai-lab/d3LLM) |
|
|
- 🌐 **Blog:** [Ultra-Fast Diffusion LLMs](https://hao-ai-lab.github.io/blogs/text-diffusion/) |
|
|
- 🕹️ **Demo:** [d3LLM Demo](https://d3llm-team.github.io/) |
|
|
|
|
|
## Model Description |
|
|
|
|
|
**d3LLM-LLaDA** is an ultra-fast diffusion language model that strikes a balance between accuracy and parallelism. It uses pseudo-trajectory distillation to teach the model which tokens can be decoded confidently at early steps, and employs an entropy-based multi-block decoding mechanism with KV-cache refresh during inference. |
|
|
|
|
|
## Key Features |
|
|
|
|
|
- 🚀 **High throughput:** 5.0× faster than autoregressive models (Qwen-2.5-7B-it) on H100 GPU and 3.5× faster on A100 GPU. |
|
|
- 📊 **High AUP:** Achieves high Accuracy Under Parallelism scores across benchmarks. |
|
|
- 🔧 **Task Optimization:** Specifically optimized for coding and math reasoning tasks. |
|
|
|
|
|
## Installation |
|
|
|
|
|
To use this model, it is recommended to clone the official repository and install the required dependencies: |
|
|
|
|
|
```bash |
|
|
# Clone the repository |
|
|
git clone https://github.com/hao-ai-lab/d3LLM.git |
|
|
cd d3LLM |
|
|
|
|
|
# Install dependencies |
|
|
pip install -r requirements.txt |
|
|
``` |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you find d3LLM useful for your research, please cite the following work: |
|
|
|
|
|
```bibtex |
|
|
@article{arxiv'26:d3llm, |
|
|
title = {d3LLM: Ultra-Fast Diffusion LLM using Pseudo-Trajectory Distillation}, |
|
|
author = {Yu-Yang Qian and Junda Su and Lanxiang Hu and Peiyuan Zhang and Zhijie Deng and Peng Zhao and Hao Zhang}, |
|
|
journal = {ArXiv preprint}, |
|
|
volume = {arXiv:2601.07568}, |
|
|
year = {2026} |
|
|
} |
|
|
``` |
