README.md

---
library_name: diffusers
pipeline_tag: image-to-image
license: other
license_name: waymo-non-commercial-and-stability-ai-community
license_link: LICENSE.md
base_model:
- stabilityai/sd-turbo
- madebyollin/taesd
tags:
- r3d2
- autonomous-driving
- image-to-image
- custom-code
- arxiv:2506.07826
---

# R3D2

One-step diffusion for realistic lighting and shadows around inserted 3D assets.
This is the **tiny VAE** version of R3D2.

[Paper](https://arxiv.org/abs/2506.07826) · [Code](https://github.com/zenseact/R3D2) · [Project](https://research.zenseact.com/publications/R3D2/) · [R3D2-big](https://huggingface.co/bertaveira/R3D2-big)

**Non-commercial use only.** This model is released under the Waymo Dataset
License Agreement. All downstream use or modification is subject to its terms,
including Section 4. Full agreements and attribution: [LICENSE.md](LICENSE.md), [NOTICE](NOTICE).

## Use with Diffusers

Tested with PyTorch 2.6.0, Diffusers 0.40.0 and Transformers 5.17.0. A CUDA GPU is required for the example below:

```bash
pip install torch==2.6.0 diffusers==0.40.0 transformers==5.17.0 peft
```

```python
import torch
from diffusers import DiffusionPipeline
from PIL import Image

pipe = DiffusionPipeline.from_pretrained(
    "bertaveira/R3D2",
    trust_remote_code=True,
    torch_dtype=torch.float16,
).to("cuda")

pipe(Image.open("input.png").convert("RGB")).images[0].save("output.png")
```

## Use with the original repository

Inside [zenseact/R3D2](https://github.com/zenseact/R3D2), after `uv sync`:

```bash
uv run huggingface-cli download bertaveira/R3D2 --local-dir checkpoints/R3D2
uv run r3d2/scripts/eval.py --model-path checkpoints/R3D2 \
    --image-path input.png --output-path output.png
```

The repository CLI uses FP32. For lower GPU memory use, use the FP16 example above.

## Credits

Built on [SD-Turbo](https://huggingface.co/stabilityai/sd-turbo) and
[TAESD](https://github.com/madebyollin/taesd), using the
[Waymo Open Dataset](https://waymo.com/open/). Powered by Stability AI.
For citation, see the [paper's repository](https://github.com/zenseact/R3D2#citation).