README.md

---
license: apache-2.0
base_model:
  - Qwen/Qwen-Image
library_name: diffusers
pipeline_tag: image-to-image
tags:
  - anima
  - qwen-image
  - vae
  - autoencoder
  - comfyui
  - anime
  - illustration
inference: false
---

# Qwanima-vae

This is a decoder-only finetune for the Qwen-Image VAE used by
[Anima](https://huggingface.co/circlestone-labs/Anima). It is directly inspired by
[spacepxl's 2x Wan VAE upscaler](https://huggingface.co/spacepxl/Wan2.1-VAE-upscale2x), adapted to
Qwen-Image and Anima using the methodology described by spacepxl. More detail about the original
method is available on that model card.

The motivation behind this project is mostly the same as spacepxl's. The Qwen-Image VAE produces a
rather unpleasant dithered look. Since it was also made with photographs in mind, it
can reconstruct fine detail as noisy or incoherent texture. This decoder finetune aims to reduce
that and produce a cleaner image suited to Anima generations.

The released checkpoint is the 45k EMA from the best 256px run. The encoder is unchanged from
Qwen-Image and only the decoder was finetuned, so this uses the same 16-channel latent format and
produces images at the same resolution as the original VAE.

## Comparison

![Original and fine-tuned VAE comparison](comparison/preview.png)

## Model

Inference is the same as with a normal VAE:

```text
Anima latent
    ↓
Qwen-Image VAE decoder (Anima finetuned)
    ↓
image
```

## Training

Training looked roughly like this:

```text
training image
    ↓
frozen Qwen VAE encoder
    ↓
clean latent
    ↓
degradation via a learned proxy model
    ↓
trainable Qwen VAE decoder
    ↓
reconstruction
```

The training data used a split of 75% Booru Essence and 25% Cleveland Museum of Art images.
Booru Essence covers the anime part of the distribution, while the museum data adds paintings,
physical media, and other fine texture to prevent the decoder from forgetting those details.

The final run used:

- 256x256 training crops
- Frozen Qwen-Image encoder
- Anima degradation proxy, applied with probability 0.8 over timesteps 0–0.12
- MSE loss at `1.0`
- DINOv3 ViT-B feature loss at `8.0`, using patch tokens from all layers
- Anti-aliased PatchGAN with LSGAN loss at `0.25`

I settled on MSE at weight 1.0 because that gave the GAN enough freedom to produce details while
still anchoring the result to the pixel target. I previously used L1, but it anchored the
reconstruction too aggressively and hampered the GAN's ability to add detail in my shoddy testing.

I won't pretend these are the optimal settings. This is simply what I ended up with in the end.

45k was selected because it looked the best. Was going to train it to 100k steps switching to 512px at 50k but it was steadily degrading as training continued. I think GAN weight might've been too high.

### Anti-aliased PatchGAN

PatchGAN judges a grid of small local patches instead of reducing the entire image to a single
real/fake score. That makes it useful for texture, but its stride-2 convolutions can also alias
high frequencies while downsampling. The discriminator can then become sensitive to pixel-grid
phase and mistake regular dithering or checkerboard patterns for convincing detail.

To make that shortcut less attractive, both the real and generated images are low-pass filtered
immediately before each of PatchGAN's three stride-2 stages. The filter is a fixed, normalized 3x3
binomial kernel:

```text
[1, 2, 1]ᵀ × [1, 2, 1] / 16
```

This band-limits the signal before subsampling, encouraging the discriminator to judge coherent
local structure instead of phase-locked pixel energy. The filtering only exists inside the
discriminator: it does not blur the saved decoder output or the images used by MSE and DINO.

The discriminator itself uses four spectral-normalized layers with 64 base channels. GAN training
began at step 1,000, ramped to weight 0.25 over 2,000 steps, and used LSGAN.

## Diffusers

`diffusion_pytorch_model.safetensors` is a standard Diffusers checkpoint. Use it with:

```python
import torch
from diffusers import AutoencoderKLQwenImage

vae = AutoencoderKLQwenImage.from_pretrained(
    "PATH_OR_REPO_ID",
    torch_dtype=torch.bfloat16,
).to("cuda")
```

## ComfyUI

A converted BF16 version for ComfyUI is located here:

```text
comfyui/qwen_vae_anima.safetensors
```

Copy it into `ComfyUI/models/vae/`, refresh or restart ComfyUI, and load it with the normal
Load VAE node.

## 2x upscaling

I mostly focused on 1x decoding for this project, so that is what I am releasing. This VAE does
not change the output resolution and a 1024px Anima latent still decodes to a 1024px image.

## Files

```text
.
├── config.json
├── diffusion_pytorch_model.safetensors
├── comfyui/
│   └── qwen_vae_anima.safetensors
├── comparison/
│   └── preview.png
└── LICENSE
```

`diffusion_pytorch_model.safetensors` is the FP32 Diffusers checkpoint.

`comfyui/qwen_vae_anima.safetensors` is the converted BF16 ComfyUI checkpoint.

## Limitations

Since this was trained mostly on anime and illustration it won't provide good results on images
far outside that distribution. Photos and textured media can be oversmoothed or otherwise
altered, and difficult high-frequency regions may develop grain or speckles. For a 2x VAE
with broader material coverage, use
[spacepxl's Wan2.1 VAE upscaler](https://huggingface.co/spacepxl/Wan2.1-VAE-upscale2x).

## License

Released under the [Apache License 2.0](LICENSE).