File size: 1,885 Bytes
2d22998 028aa20 2d22998 be9206f 2d22998 04611c0 2d22998 799a0ee 2d22998 799a0ee 2d22998 1585596 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 |
---
license: mit
datasets:
- ILSVRC/imagenet-1k
language:
- en
tags:
- diffusion
---
# Transfusion - VAE
## How to use with 🧨 diffusers
```py
from diffusers.models import AutoencoderKL
vae = AutoencoderKL.from_pretrained("lavinal712/transfusion-vae")
```
## Model
This model was trained for 50 (legacy: 7) epochs on ImageNet, COCO and FFHQ (legacy: ImageNet), with training parameters following the original Transfusion paper.
$$\mathcal{L}_{\mathrm{VAE}} = \mathcal{L}_1 + \mathcal{L}_{\mathrm{LPIPS}} + 0.5\mathcal{L}_{\mathrm{GAN}} + 0.2\mathcal{L}_{\mathrm{ID}} + 0.000001\mathcal{L}_{\mathrm{KL}}$$
## Evaluation
ImageNet 2012 (256x256, val, 50000 images)
| Model | rFID | PSNR | SSIM | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 0.408 | 28.723 | 0.845 | 0.081 |
| SD-VAE | 0.692 | 26.910 | 0.772 | 0.130 |
COCO 2017 (256x256, val, 5000 images)
| Model | rFID | PSNR | SSIM | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 2.749 | 28.556 | 0.855 | 0.078 |
| SD-VAE | 4.246 | 26.622 | 0.784 | 0.127 |
## Evaluation (legacy)
ImageNet 2012 (256x256, val, 50000 images)
| Model | rFID | PSNR | SSIM | LPIPS |
|-----------------|-------|--------|-------|-------|
| Transfusion-VAE | 0.567 | 28.195 | 0.829 | 0.100 |
| SD-VAE | 0.692 | 26.910 | 0.772 | 0.130 |
Paper: [Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model](https://arxiv.org/abs/2408.11039)
Dataset: [ImageNet](https://image-net.org/), [COCO](https://cocodataset.org/), [FFHQ](https://github.com/NVlabs/ffhq-dataset)
Base Code: [lavinal712/AutoencoderKL](https://github.com/lavinal712/AutoencoderKL)
Training Code: [lavinal712/AutoencoderKL/tree/transfusion_vae](https://github.com/lavinal712/AutoencoderKL/tree/transfusion_vae) |