speedyrulz/qwen21 / README.md
speedyrulz's picture
|
download
raw
7.13 kB
---
license: other
license_name: qwen-research
license_link: https://huggingface.co/Qwen/Qwen-Image-2.1/blob/main/LICENSE
base_model: Qwen/Qwen-Image-2.1
library_name: diffusers
tags:
- vae
- qwen-image
---
# Texture-Fix-VAE-for-Qwen-Image-2.1
<em>Built with Qwen<sup>*</sup> (an unofficial finetune of the Qwen-Image-2.1 VAE)</em>
Texture-Fix-VAE-for-Qwen-Image-2.1 is the [Qwen-Image-2.1 VAE](https://huggingface.co/Qwen/Qwen-Image-2.1/tree/main/vae), but finetuned to produce cleaner textures with no checkerboard artifacts (and no [NaNs when running in fp16](#fixing-nans-in-fp16)).
<details>
<summary>Comparison Settings</summary>
Texture-Fix-VAE-for-Qwen-Image-2.1's improved decoding is most noticeable in detailed, photo-style images.
The latents for the VAE comparison image below were generated by Qwen-Image-2.1 from the photo-style prompt:
> Landscape photograph of a subalpine wildflower meadow in the Pacific Northwest in midsummer: a clear mountain stream winding over mossy boulders through purple lupine and red paintbrush, dense old-growth Douglas fir and western red cedar forest behind, a snow-capped volcano in the distance, golden late-afternoon light, highly detailed
</details>
| Qwen-Image-2.1-VAE ([full-res](./images/decode-qwen-image-2.1-vae-full.png)) | 🪄 Texture-Fix-VAE-for-Qwen-Image-2.1 🪄 ([full-res](./images/decode-texture-fix-vae-full.png)) |
| --- | --- |
| ![](./images/decode-qwen-image-2.1-vae-zoomed-1.png) | ![](./images/decode-texture-fix-vae-zoomed-1.png) |
| ![](./images/decode-qwen-image-2.1-vae-zoomed-2.png) | ![](./images/decode-texture-fix-vae-zoomed-2.png) |
| ![](./images/decode-qwen-image-2.1-vae-full.png) | ![](./images/decode-texture-fix-vae-full.png) |
# Usage
**ComfyUI**
Download [`texture_fix_vae_for_qwen_image_2.1_bf16.safetensors`](./texture_fix_vae_for_qwen_image_2.1_bf16.safetensors) into `ComfyUI/models/vae/` and select it in the `Load VAE` node, in place of `qwen_image_2.1_vae_bf16.safetensors`. It also works with `--fp16-vae`.
**🧨 Diffusers**
```python
import torch
from diffusers import QwenImage21Pipeline, AutoencoderKLQwenImage21
vae = AutoencoderKLQwenImage21.from_pretrained("madebyollin/texture-fix-vae-for-qwen-image-2.1", torch_dtype=torch.bfloat16)
pipe = QwenImage21Pipeline.from_pretrained("Qwen/Qwen-Image-2.1", vae=vae, torch_dtype=torch.bfloat16).to("cuda")
```
# Mechanism
## Fixing Checkerboard Artifacts
Texture-Fix-VAE-for-Qwen-Image-2.1 was cured of checkerboard artifacts by finetuning the highest-resolution blocks briefly using the adversarial recipe developed for [TAESD](https://github.com/madebyollin/taesd).
The TAESD recipe, like most image autoencoder training recipes, uses a mix of PSNR-focused (MSE/MAE), LPIPS, and adversarial (GAN) loss terms.
Whenever precise details can't be reconstructed, MSE/MAE loss encourages blurring, LPIPS loss encourages blurring+checkerboarding (among other artifacts), and adversarial loss encourages generating sharp/plausible (but fake) detail without obvious artifacts.
This figure from [DC-AE](https://arxiv.org/abs/2410.10733) shows the importance of including adversarial (GAN) loss:
![Demo of the effects of adversarial loss, courtesy of the DC-AE paper](https://cdn-uploads.huggingface.co/production/uploads/630447d40547362a22a969a2/uj9vJSR34LyZEkQeL-VOU.png)
<sup>Figure: from *Deep Compression Autoencoder for Efficient High-Resolution Diffusion Models* (Chen et al., 2024, [arXiv:2410.10733](https://arxiv.org/abs/2410.10733)), licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/); cropped to the first two rows.</sup>
I suspect the original Qwen-Image-2.1-VAE was trained without a working adversarial loss term.
## Fixing NaNs in FP16
Texture-Fix-VAE-for-Qwen-Image-2.1 runs fine in fp16, whereas the original Qwen-Image-2.1 VAE decoder often produces NaNs (the 🟪 magenta-highlighted regions below):
| Qwen-Image-2.1-VAE (bf16) | ⚠️ Qwen-Image-2.1-VAE (fp16) | Texture-Fix-VAE-for-Qwen-Image-2.1 (fp16) |
| --- | --- | --- |
| ![](./images/fp16-demo-official-bf16.png) | ![](./images/fp16-demo-official-fp16.png) | ![](./images/fp16-demo-texture-fix-fp16.png) |
<sup>The original VAE decoder mostly produces NaNs in fully-transparent regions, but they bleed into surrounding content. You can reproduce this visualization with [`fp16_demo.py`](./fp16_demo.py).</sup>
Texture-Fix-VAE-for-Qwen-Image-2.1 was cured of the NaNs-in-FP16 issue by:
1) Finetuning the decoder weights to keep outputs the same while reducing activation magnitudes (like [SDXL-VAE-FP16-Fix](https://huggingface.co/madebyollin/sdxl-vae-fp16-fix))
2) Manually shrinking certain weights to further reduce the scale of the residual stream (every residual branch starts with an RMSNorm, so this doesn't change the outputs)
# Metrics
Texture-Fix-VAE-for-Qwen-Image-2.1 makes perceptual quality metrics (rFID) better and reconstruction accuracy metrics (LPIPS/PSNR) slightly worse.
| Metric | Qwen-Image-2.1-VAE (bf16) | Texture-Fix-VAE-for-Qwen-Image-2.1 (bf16) | Texture-Fix-VAE-for-Qwen-Image-2.1 (fp16) |
| --- | --- | --- | --- |
| rFID ↓ (COCO val2017, 5000 images @ 256²) | 3.38 | 2.07 | **2.06** |
| PSNR ↑ (COCO val2017 @ 256²) | **33.29** | 32.73 | 32.79 |
| LPIPS ↓ (COCO val2017 @ 256²) | **0.0357** | 0.0384 | 0.0380 |
| PSNR ↑ (DIV2K valid, native 1024² crops) | **32.85** | 32.35 | 32.40 |
| LPIPS ↓ (DIV2K valid, native 1024² crops) | **0.0460** | 0.0490 | 0.0486 |
Texture-Fix-VAE-for-Qwen-Image-2.1 also reduces the internal activation magnitude, preventing NaNs/overflows in fp16.
| fp16 Metric (encoder + decoder in fp16) | Qwen-Image-2.1-VAE | Texture-Fix-VAE-for-Qwen-Image-2.1 |
| --- | --- | --- |
| Largest decoder activation ↓ (fp16 max is 65504) | ~3.5M (overflows) | **~1.3k** |
| Inputs with NaN outputs ↓ (185 stress-test inputs: transparent / opaque / synthetic images and random latents, 256² to 2048²) | 75 | **0** |
# Version History
- 2026-10-05 Updated release; now also fixes decoder NaNs in fp16, metrics are similar
- Finetuned the decoder like SDXL-VAE-FP16-Fix (scale + bias of each conv and the RMSNorm gains, 61k parameters) to match the initial release while penalizing large activations, with extra transparent training crops (~60k steps)
- Re-ran the initial release's finetune (8000 steps) to restore texture detail
- Rescaled the weights by exact powers of 2 (decoder /512, encoder /8)
- 2026-09-24 Initial release; finetuned decoder to fix the checkerboard artifacts
- ~5000 steps at learning rate 3e-5, with only the two highest-resolution decoder stages and output head unfrozen (7.5M trainable parameters)
# Attribution Notice
This fine-tuned VAE is based on Qwen/Qwen-Image-2.1; original materials © 2026 Hangzhou Tongyi Laboratory Technology Co., Ltd., licensed under the Qwen RESEARCH LICENSE AGREEMENT (see [LICENSE](./LICENSE)) for non-commercial/research use only. Built with Qwen<sup>*</sup>.
<sup>* In the sense that the **initial VAE weights** are from Qwen-Image. The decoder fine-tuning work was performed by `madebyollin` and Claude Opus.</sup>

Xet Storage Details

Size:
7.13 kB
·
Xet hash:
5432492954ffe9cdd37146e7391305e0137e0b6d250505988297802616b8c895

Xet efficiently stores files, intelligently splitting them into unique chunks and accelerating uploads and downloads. More info.