| license: apache-2.0 | |
| base_model: Qwen/Qwen3-4B | |
| pipeline_tag: text-generation | |
| # Qwen3-4B-A2D-untrained-dllm-convert | |
| This repository contains the Qwen3-4B model converted to the A2D architecture (bidirectional attention), as presented in the paper [Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation](https://huggingface.co/papers/2606.06712). | |
| This specific artifact serves as an **untrained student initialization** for the On-Policy Distillation (OPD) process to transform an autoregressive model into a diffusion language model. | |
| - **Project Page:** [https://opdlm.vercel.app/](https://opdlm.vercel.app/) | |
| - **GitHub Repository:** [https://github.com/divelab/OPDLM](https://github.com/divelab/OPDLM) | |
| - **Base model**: [Qwen/Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B) | |
| ## Model Details | |
| - **Architecture**: A2D-Qwen3 (non-causal attention, same weights as original) | |
| - **Parameters**: 4.02B | |
| - **Vocab size**: 151936 | |
| - **Model type**: `a2d-qwen3` | |
| This model has the original Qwen3-4B weights with bidirectional (non-causal) attention. It was converted using the [dllm convert pipeline](https://github.com/ZHZisZZ/dllm/blob/b8d76ff74b2053d359cd88fedfbc6362db17e3d7/examples/a2d/README.md?plain=1#L49-L53). No diffusion pretraining or SFT has been applied. | |
| **Mask token registration**: The mask token `<|MASK|>` (ID 151669) is registered in the tokenizer for use with diffusion-based language modeling. The original Qwen3 tokenizer includes `<|MASK|>` in `special_tokens_map.json` but does not register it in `tokenizer_config.json`, so `tokenizer.mask_token_id` returns `None`. We fixed this by adding `<|MASK|>` to the `added_tokens_decoder` section and the `mask_token` field in `tokenizer_config.json`, and adding the full `mask_token` entry in `special_tokens_map.json`. After this fix, `tokenizer.mask_token_id` correctly returns `151669`. | |
| ## Citation | |
| ```bibtex | |
| @misc{su2026opdlm, | |
| title={Data-Efficient Autoregressive-to-Diffusion Language Models via On-Policy Distillation}, | |
| author={Xingyu Su and Jacob Helwig and Shubham Parashar and Atharv Chagi and Lakshmi Jotsna and Degui Zhi and James Caverlee and Dileep Kalathil and Shuiwang Ji}, | |
| year={2026}, | |
| eprint={2606.06712}, | |
| archivePrefix={arXiv}, | |
| primaryClass={cs.CL}, | |
| url={https://arxiv.org/abs/2606.06712}, | |
| } | |
| ``` |