File size: 1,989 Bytes
e6ef705 5e5f00e e6ef705 5e5f00e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 | ---
license: mit
language:
- en
tags:
- DLLM
- diffusion-language-model
- on-policy-distillation
- post-training
library_name: transformers
pipeline_tag: text-generation
base_model: Qwen/Qwen3-4B
datasets:
- divelab/opdlm_train_data
arxiv: 2606.06712
---
# OPDLM-4B
OPDLM-4B is a block diffusion language model (DLM) obtained by post-training an
autoregressive language model (ARLM) into a diffusion language model via
**on-policy distillation**. arXiv report: [arxiv.org/abs/2606.06712](https://arxiv.org/abs/2606.06712)
## Highlights
- **Converted, not pretrained from scratch:** built from a strong ARLM, reusing its prior.
- **Training-efficient:** ~0.076B tokens of conversion vs. ~50B tokens for from-scratch DLM training (same base ARLM).
- **Inference-efficient:** parallel token decoding via block diffusion.
## Model Details
- **Developed by:** DIVE Lab, Texas A&M University
- **Base model:** [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B)
- **Model type:** Block diffusion language model (decoder-based)
- **Block size:** 4
- **Parameters:** ~4B
- **Language:** English
- **License:** MIT
## Training
- **Method:** On-policy distillation from a frozen ARLM teacher into a block DLM student.
- **Conversion budget:** ~0.076B tokens
- **Data:** [opdlm_train_data](https://huggingface.co/datasets/divelab/opdlm_train_data)
## Results
For detailed results and benchmarks, please refer to our paper: [arxiv.org/abs/2606.06712](https://arxiv.org/abs/2606.06712)
## Citation
```bibtex
@misc{su2026dataefficientautoregressivetodiffusionlanguagemodels,
title={Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation},
author={Xingyu Su and Jacob Helwig and Shubham Parashar and Atharv Chagi and Lakshmi Jotsna and Degui Zhi and James Caverlee and Dileep Kalathil and Shuiwang Ji},
year={2026},
eprint={2606.06712},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2606.06712},
}
``` |