README.md

---
license: other
license_name: nvidia-license
license_link: LICENSE
---

# Model Overview


## Description

NL-Diffusion-Image introduces a new paradigm for high-resolution text-to-image generation via LLM based on masked discrete diffusion over tokenized image patches.
Each image is encoded into a sequence of discrete tokens (using a 128K codebook/vocabulary), and generation proceeds through iterative parallel unmasking - similar to Diffusion LLMs.
We finetune from [Nemotron-Labs-Diffusion](https://huggingface.co/nvidia/Nemotron-Labs-Diffusion-8B) and introduce 2 key components:

* A token-editing mechanism that allows the model to revise already-unmasked tokens during inference.
* Grouped Cross-Entropy (GCE) objective to handle large-vocabulary training efficiently.

This model is ready for research or non-commercial evaluation.


![Teaser Image](assets/teaser.png)
### License/Terms of Use

GOVERNING TERMS: Use of this model is governed by the [NVIDIA Open Source License Agreement](https://huggingface.co/nvidia/NL-Diffusion-Image/raw/main/LICENSE).

## Deployment Geography

Global

## Use Case

This model is intended for text-to-image generation.

## Release Date

Hugging Face: 07/01/2026 via [HuggingFace](https://huggingface.co/nvidia/NL-Diffusion-Image).

## References

* NL-Diffusion-Image Paper: Shufan Li et al., "Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis,".
* Nemotron-Labs-Diffusion Paper: Yonggan Fu et al., "Nemotron-Labs-Diffusion: A Tri-Mode Language Model Unifying Autoregressive, Diffusion, and Self-Speculation Decoding".
* Emu3.5 Paper: Emu3.5 team, "Emu3.5: Native Multimodal Models are World Learners".

## Model Architecture

**Architecture Type:** Neural Network <br>
**Network Architecture:** Masked Diffusion Transformer, IBQ tokenizer for visual encoding/decoding <br>
**Number of model parameters:** ~8B parameters <br>

We encode 16x16 image patches using a pretrained discrete tokenizer from Emu3.5, with a codebook size of 128k token IDs.
We expand the Nemotron-Labs-Diffusion vocabulary with a corresponding number of randomly-initialized embeddings, and fine-tune the model on a dataset of image/caption pairs.

## Input

**Input Type(s):** Text <br>
**Input Format(s):** Characters <br>
**Other Properties Related to Input:** Maximum prompt length is 900 tokens. <br>

## Output

**Output Type(s):** Images <br>
**Output Format:** Tensor (3xHxW) <br>
**Other Properties Related to Output:** The output represents an RGB image. <br>

## Software Integration

**Runtime Engine(s):**
* PyTorch <br>


**Supported Hardware Microarchitecture Compatibility:** <br>
* NVIDIA Ampere <br>
* NVIDIA Blackwell <br>
* NVIDIA Jetson <br>
* NVIDIA Hopper <br>
* NVIDIA Lovelace <br>
* NVIDIA Pascal <br>
* NVIDIA Turing <br>
* NVIDIA Volta <br>

**[Preferred/Supported] Operating System(s):** <br>
* Linux
* Linux 4 Tegra
* QNX
* Windows

The integration of foundation and fine-tuned models into AI systems requires additional testing using use-case-specific data to ensure safe and effective deployment.
Following the V-model methodology, iterative testing and validation at both unit and system levels are essential to mitigate risks, meet technical and functional requirements, and ensure compliance with safety and ethical standards before deployment.

This AI model can be embedded as an Application Programming Interface (API) call into the software environment described above.

## Model Version(s)

* NL-Diffusion-Image (8B).

**Links:**

* https://huggingface.co/nvidia/NL-Diffusion-Image

# Training and Evaluation Datasets

## Training Dataset

**LAION-115M-Clean Recaptioned**

**Data Modality:**
* Multimodal (image, caption)

**Image Training Data Size:**
* 115M samples

**Data Collection Method by dataset:**
* Web scraping

**Labeling Method by dataset:**
* Subset of 8M images recaptioned using Qwen3-VL

**MidJourney v6 520k Recaptioned**

**Data Modality:**
* Multimodal (image, caption)

**Image Training Data Size:**
* 520k samples

**Data Collection Method by dataset:**
* Automated

**Labeling Method by dataset:**
* Images recaptioned using Qwen3-VL

**COYO700M Recaptioned**

**Data Modality:**
* Multimodal (image, caption)

**Image Training Data Size:**
* 700M samples

**Data Collection Method by dataset:**
* Automated

**Labeling Method by dataset:**
* Subset of 24M images recaptioned using Qwen3-VL

**BLIP3o-60k Recaptioned**

**Data Modality:**
* Multimodal (image, caption)

**Image Training Data Size:**
* 520k samples

**Data Collection Method by dataset:**
* Automated

**Labeling Method by dataset:**
* Images recaptioned using Qwen3-VL

## Evaluation Datasets

**ImageNet**

**Link:**
* [ImageNet](https://www.image-net.org/)

**Data Collection:**
* Automated

**Labeling Method:**
* Manually-Collected

**Training Images:**
* 1,281,167

**Validation Images:**
* 50,000

**GenEval**

**Link:**
* [GenEval](https://github.com/djghosh13/geneval)

**Data Collection:**
* Manually-Collected

**Labeling Method:**
* Manually-Collected

**Captions/annotations:**
* 553 samples

**DPGBench**

**Link:**
* [DPGBench](https://github.com/TencentQQGYLab/ELLA)

**Data Collection:**
* Manually-Collected

**Labeling Method:**
* Manually-Collected

**Captions/annotations:**
* 1065 samples

**MJHQ-30K**

**Link:**
* [MJHQ-30K](https://huggingface.co/datasets/playgroundai/MJHQ-30K/blob/main/README.md)

**Data Collection:**
* Manually-Collected

**Labeling Method:**
* Automated

**CaptionsImages:**
* 30k samples

## GenEval Benchmark


| Model      | Params | Single Object | Two Objects | Counting | Colors | Position | Color Attri. | Overall |
|------------|---------|-------------|-----------|-------------|--------|----------|--------------|---------|
| Qwen-Image-2507 | 20B | 0.99 | 0.92 | 0.89 | 0.88 | 0.76 | 0.77 | 0.87 |
| Nemotron-Labs-Diffusion-Image | 8B | 0.98 | 0.93 | 0.83 | 0.94 | 0.88 | 0.82 | 0.90 |

## Text-to-Image Generation Performance on DPG Benchmark and MJHQ-30k Dataset

| Model      | Params | Codebook | DPG | MJHQ FID | MJHQ  HPSv3 |
|------------|--------|-----------|----|----------|-------------|
| MMaDa      | 8B     | 8,192 | 53.4 | 32.85 | 5.43 |
| LaViDa-O   | 10B    | 8,192 | 81.8 | 6.68 | 8.81 |
| Nemotron-Labs-Diffusion-Image | 8B | 131,072 | 85.2 | 6.46 | 9.61|
| Nemotron-Labs-Diffusion-Image*</sup>  | 8B | 131,072 | 86.9 | 12.23 | 10.76 |

<sup>*</sup> Finetuned on 6M synthetic data for better image quality


## Inference

**Acceleration Engine:** TensorRT, TensorRT-LLM <br>
**Engine:** PyTorch <br>
**Test Hardware:** NVIDIA Hopper (H100) <br>

## Ethical Considerations

NVIDIA believes Trustworthy AI is a shared responsibility and we have established policies and practices to enable development for a wide array of AI applications. Developers should work with their internal model team to ensure this model meets requirements for the relevant industry and use case and addresses unforeseen product misuse.

Please make sure you have proper rights and permissions for all input image and video content; if image or video includes people, personal health information, or intellectual property, the image or video generated will not blur or maintain proportions of image subjects included.

For more detailed information on ethical considerations for this model, please see the Model Card++ Explainability, Bias, Safety & Security, and Privacy Subcards below.

Please report model quality, risk, security vulnerabilities or NVIDIA AI Concerns [here](https://www.nvidia.com/en-us/support/submit-security-vulnerability/).

### Bias

Field                                                                                               |  Response
:---------------------------------------------------------------------------------------------------|:---------------
Participation considerations from adversely impacted groups [protected classes](https://www.senate.ca.gov/content/protected-classes) in model design and testing:  |  None
Measures taken to mitigate against unwanted bias:                                                   |  None
Bias Metric (If Measured): | None

### Explainability

Field                                                                                                  |  Response
:------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------
Intended Task/Domain:                                                                                  |  Text-to-image generation
Model Type:                                                                                            |  Masked Diffusion Model
Intended Users:                                                                                        |  Research
Output:                                                                                                |  Images
Describe how the model works:                                                                          |  The model takes a caption as input and generated an image.
Name the adversely impacted groups this has been tested to deliver comparable outcomes regardless of:  |  Not Applicable
Technical Limitations:                                                                                 |  The model generates images in a single resolution of 1024x1024 pixels.
Verified to have met prescribed NVIDIA quality standards:  |  Yes
Performance Metrics:                                                                                   |  GenEVal, DPG, MJHQ.
Potential Known Risks:                                                                                 |  This model may not perform well on visual domains that are not represented in the training data. The generated images might fail to disambiguate differences in prompts that appear evident to humans. Domain-specific evaluation is required for the target application.
Licensing:                                                                                             |  [NVIDIA Open Source License](https://huggingface.co/nvidia/NL-Diffusion-Image/raw/main/LICENSE)


### Privacy

Field                                                                                                                              |  Response
:----------------------------------------------------------------------------------------------------------------------------------|:-----------------------------------------------
Generatable or reverse engineerable personal data?                                                                               |  No
Personal data used to create this model?                                                                                       |  No
How often is dataset reviewed?                                                                                                     |  Before Every Release
Is there provenance for all datasets used in training?                                                                                |  Yes
Does data labeling (annotation, metadata) comply with privacy laws?                                                                |  Yes
Is data compliant with data subject requests for data correction or removal, if such a request was made?                           |  Yes
Was data from user interactions with the AI model (e.g. user input and prompts) used to train the model?                           | No
Applicable Privacy Policy                                                                          | https://www.nvidia.com/en-us/about-nvidia/privacy-policy/

### Safety

Field                                               |  Response
:---------------------------------------------------|:----------------------------------
Model Application Field(s):                               |  Generation of images
Describe the life critical impact (if present).   |  Not Applicable
Use Case Restrictions:                              |  Research/evaluation only, non-commercial applications.
Model and dataset restrictions:            |  The Principle of least privilege (PoLP) is applied limiting access for dataset generation and model development. Restrictions enforce dataset access during training, and dataset license constraints adhered to.


## Quick Start 

```
import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

model_path = "shufanlNvidia/NLD-Diffusion-Image-8B-Internal"
tokenizer = PreTrainedTokenizerFast.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
)
model = model.to("cuda").eval()
model.config.dlm_paradigm = "bidirectional"
PROMPT = (
    "A full-body shot of hyper-realistic female cyborg, human facial skin seamlessly integrated with a glossy white mechanical head shell. "
    "Features a realistic human ear, blue eyes. bright, outdoor, background with blue sky, illuminated by striking bright white studio lighting, "
    "casting soft shadows. Cyberpunk aesthetic, high-tech minimalism, shot on 85mm lens, photorealistic, Unreal Engine 5 render, intricately detailed, "
    "8k resolution, high dynamic range, chest with whit armor plate, cute, beautiful, sexy, glossy surface, reflective, Artstation, pixiv, no hair, "
    "3D render, stylized eyesz"
)
torch.manual_seed(42)

image = model.text_to_image(
    PROMPT,
    tokenizer=tokenizer,
    image_resolution=1024,
    n_tokens=(1024 // 16) * (1024 // 16),
    guidance_scale=5.0,
    temperature=0.86,
    n_steps=64,
    schedule="shift",
    shift=5,
    confidence_policy="mmada",
    schedule_temp="linear",
    alg_temp=1.0,
    dynamic_temperature=False,
    min_temperature=0.01,
    edit_threshold=0.6,
    micro_cond="ORIGINAL WIDTH : 1024; ORIGINAL HEIGHT : 1024; TOP : 0; LEFT : 0; SCORE : 6.520; HPS: 3.220",
    block_policy=2,
    is_legacy=False,
    use_cache=False,
)
image.save("output.webp")
```

See [demo_inference_release.py](demo_inference_release.py) for more details of inference arguments.