README.md

---
license: apache-2.0
library_name: transformers
tags:
- dllm
- diffusion
- llm
- text_generation
---

# LLaDA2.2-mini

**LLaDA2.2-mini** is the lightweight variant of the agentic diffusion language model in the LLaDA2 series. Built upon the LLaDA2.0-mini architecture, it inherits the core innovations of the LLaDA2.2 series — **Levenshtein Editing** (introducing `DELETE` and `INSERT` control tokens) — enabling long-context tool calling, multi-turn interaction, and robust error correction, while maintaining a smaller parameter footprint and lower inference cost. For more details, please refer to our [technical report](https://github.com/inclusionAI/LLaDA2.X/blob/main/LLaDA2_2_tech_report.pdf).

---

## 📊 Benchmarks

The following table compares **LLaDA2.0-mini**, **LLaDA2.1-mini**, and **LLaDA2.2-mini** across General and Agentic capabilities.

<table>
<thead>
<tr>
<th align="center">Category</th>
<th align="left">Benchmark</th>
<th align="center">LLaDA2.0-mini</th>
<th align="center">LLaDA2.1-mini</th>
<th align="center">LLaDA2.2-mini</th>
</tr>
</thead>
<tbody>
<tr><td colspan="5" align="center" bgcolor="#E8F0FE"><b>General</b></td></tr>
<tr><td align="center" rowspan="2">Function Calling</td><td>BFCL v4</td><td align="center">25.05</td><td align="center">28.44</td><td align="center">47.68</td></tr>
<tr><td>BFCL v3</td><td align="center">70.72</td><td align="center">72.06</td><td align="center">69.02</td></tr>
<tr><td align="center" rowspan="2">Math</td><td>AIME 2026</td><td align="center">37.71</td><td align="center">40.37</td><td align="center">35.05</td></tr>
<tr><td>OlympiadBench</td><td align="center">67.70</td><td align="center">64.30</td><td align="center">61.11</td></tr>
<tr><td align="center" rowspan="2">Coding</td><td>LiveCodeBench v6</td><td align="center">27.70</td><td align="center">28.80</td><td align="center">28.14</td></tr>
<tr><td>MultiPL-E</td><td align="center">67.46</td><td align="center">64.16</td><td align="center">65.26</td></tr>
<tr><td align="center" rowspan="2">Instruction Following</td><td>IFBench</td><td align="center">32.33</td><td align="center">31.60</td><td align="center">24.93</td></tr>
<tr><td>Multi-IF</td><td align="center">60.58</td><td align="center">58.43</td><td align="center">57.03</td></tr>
<tr><td align="center">Reasoning</td><td>KOR-Bench</td><td align="center">49.92</td><td align="center">46.64</td><td align="center">43.60</td></tr>
<tr><td align="center">Knowledge</td><td>GPQA-Diamond</td><td align="center">47.76</td><td align="center">48.36</td><td align="center">44.41</td></tr>
<tr><td align="center">Long Context</td><td>LongBench v2</td><td align="center">15.51</td><td align="center">12.13</td><td align="center">34.99</td></tr>
<tr><td colspan="2" align="center"><b>General Average</b></td><td align="center"><b>45.68</b></td><td align="center"><b>45.03</b></td><td align="center"><b>46.47</b></td></tr>
<tr><td colspan="5" align="center" bgcolor="#FEF3E2"><b>Agentic</b></td></tr>
<tr><td align="center" rowspan="3">Agent</td><td>τ²-Bench</td><td align="center">-</td><td align="center">-</td><td align="center">57.50</td></tr>
<tr><td>Claw-Eval</td><td align="center">-</td><td align="center">-</td><td align="center">57.16</td></tr>
<tr><td>PinchBench</td><td align="center">-</td><td align="center">-</td><td align="center">62.33</td></tr>
<tr><td colspan="2" align="center"><b>Agentic Average</b></td><td align="center"><b>-</b></td><td align="center"><b>-</b></td><td align="center"><b>59.00</b></td></tr>
</tbody>
</table>

---

## 🚀 Key Features

+ **Efficient 128K Diffusion Infrastructure**: LLaDA2.2-mini extends the context window to **128K** and introduces the **Block Routing** mechanism, which restricts MoE expert activation at the diffusion block level, enabling efficient long-context agentic tasks.

+ **Levenshtein Editing**: Introduces **DELETE** and **INSERT** control tokens, enabling diffusion decoding to edit sequence structure, remove redundant content, and create insertion points during parallel generation.

+ **Agentic Reinforcement Learning**: Proposes **Levenshtein Editing ELBO-based Block-level Policy Optimization (L-EBPO)**, leveraging agentic environment rewards to train Levenshtein editing and error correction capabilities in multi-turn tool-use scenarios.

+ **Lightweight & Efficient**: With a total of **16B** parameters and only **1.4B** activated during inference, it significantly reduces computational cost while maintaining strong capabilities.

---

## 📦 Model Variants

| Model ID | Description | Hugging Face Link |
| --- | --- | --- |
| `inclusionAI/LLaDA2.2-flash` | Agentic MoE Diffusion Language Model (100B) with Levenshtein editing capabilities. | [🤗 Model Card](https://huggingface.co/inclusionAI/LLaDA2.2-flash) |
| `inclusionAI/LLaDA2.2-mini` | Lightweight Agentic MoE Diffusion Language Model (16B) with Levenshtein editing capabilities. | [🤗 Model Card](https://huggingface.co/inclusionAI/LLaDA2.2-mini) |

---

## 🔍 Model Overview

Key specifications of **LLaDA2.2-mini**:

+ **Type**: Mixture-of-Experts (MoE) Diffusion Language Model with Levenshtein Editing
+ **Context Length**: 128K tokens
+ **Levenshtein Editing Control Tokens**: `DELETE`, `INSERT`
+ **Total Parameters (excl. Embedding)**: 16B
+ **Layers**: 20
+ **Attention Heads**: 16
+ **KV Heads**: 4
+ **Experts**: 256 (8 activated per token)
+ **Positional Encoding**: Rotary Position Embedding (RoPE)
+ **Vocabulary Size**: 157,184

---

## 🤗 Hugging Face Transformers Usage

Please ensure `transformers>=5.2.0` and related dependencies are installed.

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = "inclusionAI/LLaDA2.2-mini"

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
)
model = model.to(torch.bfloat16)
model.eval()

tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)

prompt = "Calculate 1+5-28*0.5-200=?"
input_ids = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    add_generation_prompt=True,
    tokenize=True,
    return_tensors="pt",
).input_ids

generated_tokens = model.generate(
    inputs=input_ids,
    eos_early_stop=True,
    gen_length=512,
    block_length=32,
    threshold=0.5,
    editing_threshold=0.0,
    temperature=0.0,
)

generated_answer = tokenizer.decode(
    generated_tokens[0],
    skip_special_tokens=True,
)
print(generated_answer)
```

### Best Practices

For optimal performance, we recommend the following configurations:

1. **Sampling Parameters**: Use `block_length=32`, `temperature=0.0`, `top_p=None`, `top_k=None` as stable defaults.

2. **Denoising Threshold**: Adjust `threshold`, `editing_threshold`, and `max_post_steps` based on the speed-quality trade-off for your use case. Lower thresholds can improve inference speed but may lead to repetitive or unstable outputs.

3. **Output Length**: For most queries, an output length of 32768 tokens is recommended.

4. **Long-Context Agentic Tasks**: For long-context tool calling and multi-turn agentic applications, we recommend using **SGLang** as the serving backend. Ensure the server configuration supports a 128K context window and the model's MoE diffusion inference requirements.

---

## 🤖 ModelScope

If you are in mainland China, we strongly recommend accessing our models via 🤖 [ModelScope](https://modelscope.cn/models/inclusionAI/LLaDA2.2-mini).

---

## 🌐 License

This project is licensed under the [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0).

---

## 🤝 Contact & Collaboration

For any questions, collaboration opportunities, or feedback, please reach out to us via [Hugging Face](https://huggingface.co/inclusionAI/LLaDA2.2-mini) or submit an issue on our [GitHub repository](https://github.com/inclusionAI).

Join us in advancing open, efficient, and intelligent diffusion language models for agentic applications!

---