---
license: mit
datasets:
- ILSVRC/imagenet-1k
language:
- en
tags:
- diffusion
---

# Transfusion - VAE

## How to use with 🧨 diffusers

```py
from diffusers.models import AutoencoderKL
vae = AutoencoderKL.from_pretrained("lavinal712/transfusion-vae")
```

## Model 

This model was trained for 50 (legacy: 7) epochs on ImageNet, COCO and FFHQ (legacy: ImageNet), with training parameters following the original Transfusion paper.

$$\mathcal{L}_{\mathrm{VAE}} = \mathcal{L}_1 + \mathcal{L}_{\mathrm{LPIPS}} + 0.5\mathcal{L}_{\mathrm{GAN}} + 0.2\mathcal{L}_{\mathrm{ID}} + 0.000001\mathcal{L}_{\mathrm{KL}}$$

## Evaluation

ImageNet 2012 (256x256, val, 50000 images)

| Model           | rFID  | PSNR   | SSIM  | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 0.408 | 28.723 | 0.845 | 0.081 |
| SD-VAE          | 0.692 | 26.910 | 0.772 | 0.130 |

COCO 2017 (256x256, val, 5000 images)

| Model           | rFID  | PSNR   | SSIM  | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 2.749 | 28.556 | 0.855 | 0.078 |
| SD-VAE          | 4.246 | 26.622 | 0.784 | 0.127 |

## Evaluation (legacy)

ImageNet 2012 (256x256, val, 50000 images)

| Model           | rFID  | PSNR   | SSIM  | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 0.567 | 28.195 | 0.829 | 0.100 |
| SD-VAE          | 0.692 | 26.910 | 0.772 | 0.130 |

Paper: [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://arxiv.org/abs/2408.11039)

Dataset: [ImageNet](https://image-net.org/), [COCO](https://cocodataset.org/), [FFHQ](https://github.com/NVlabs/ffhq-dataset)

Base Code: [lavinal712/AutoencoderKL](https://github.com/lavinal712/AutoencoderKL)

Training Code: [lavinal712/AutoencoderKL/tree/transfusion_vae](https://github.com/lavinal712/AutoencoderKL/tree/transfusion_vae)