---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
---

# iLLaDA-8B-Instruct

iLLaDA is an 8B fully bidirectional masked diffusion language model trained from scratch with 12T pre-training tokens, an 8192-token context length, variable-length generation, and confidence-based scoring for multiple-choice evaluation.

For more details, please refer to the paper: [Improved Large Language Diffusion Models](https://huggingface.co/papers/2606.25331).

Inference and evaluation codes can be found in the [LLaDA GitHub Repository](https://github.com/ML-GSAI/LLaDA).

## How to Use

You can load the model and tokenizer using the `transformers` library:

```python
import torch
from transformers import AutoModel, AutoTokenizer

# Load the tokenizer and model
tokenizer = AutoTokenizer.from_pretrained('GSAI-ML/iLLaDA-8B-Instruct', trust_remote_code=True)
model = AutoModel.from_pretrained('GSAI-ML/iLLaDA-8B-Instruct', trust_remote_code=True, torch_dtype=torch.bfloat16)
```

For customized generation and evaluation scripts (such as `generate.py` and `chat.py`), please visit the official [GitHub repository](https://github.com/ML-GSAI/LLaDA).

## Architecture

|  | iLLaDA 8B | LLaDA 8B |
| --- | ---: | ---: |
| Layers | 32 | 32 |
| Model dimension | 4096 | 4096 |
| Attention heads | 32 | 32 |
| Key/Value heads | 8 | 32 |
| FFN dimension | 14,336 | 12,288 |
| Vocabulary size | 155,136 | 126,464 |
| Maximum sequence length | 8192 | 4096 |
| Embedding and LM-head | Tied | Untied |
| Total parameters | 7.62B | 8.02B |
| Non-embedding parameters | 6.98B | 6.98B |

## Benchmark Results of Instruct Models

|  | iLLaDA 8B | LLaDA 8B | Dream 7B | Qwen2.5 7B |
| --- | ---: | ---: | ---: | ---: |
| Model | Diffusion | Diffusion | Diffusion | AR |
| MMLU | 71.6 | 65.5 | 67.0 | 76.6 |
| MMLU-Pro | 52.3 | 37.0 | 43.3 | 56.3 |
| MMLU-Redux | 76.4 | 68.9 | 76.3 | 75.7 |
| GSM8K | 89.0 | 77.5 | 81.0 | 91.6 |
| MATH | 56.7 | 42.2 | 39.2 | 75.5 |
| HumanEval | 65.9 | 49.4 | 55.5 | 84.8 |
| MBPP | 58.0 | 41.0 | 58.8 | 79.2 |
| Average | 67.1 | 54.5 | 60.2 | 77.1 |

## Citation

```bibtex
@article{nie2025large,
  title={Large Language Diffusion Models},
  author={Nie, Shen and Zhu, Fengqi and You, Zebin and Zhang, Xiaolu and Ou, Jingyang and Hu, Jun and Zhou, Jun and Lin, Yankai and Wen, Ji-Rong and Li, Chongxuan},
  journal={arXiv preprint arXiv:2502.09992},
  year={2025}
}
```