File size: 3,810 Bytes
e51b0dd | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 | ---
license: apache-2.0
library_name: diffusers
pipeline_tag: unconditional-image-generation
tags:
- diffusers
- fit
- image-generation
- class-conditional
- imagenet
inference: true
widget:
- output:
url: FiTv1-XL-2-256/demo.png
language:
- en
---
# FiT-diffusers
Diffusers-ready checkpoints for **Flexible Vision Transformer (FiT)** and **FiTv2**, converted from [`InfImagine/FiT`](https://huggingface.co/InfImagine/FiT) / [`InfImagine/FiTv2`](https://huggingface.co/InfImagine/FiTv2).
> **Re-distribution notice:** weights and configs in this repo are re-packaged from the official InfImagine releases. Original work: [FiT (arXiv:2402.12376)](https://arxiv.org/pdf/2402.12376.pdf), [FiTv2 (arXiv:2410.13925)](https://arxiv.org/pdf/2410.13925). License: [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).
This repo is derived from the development bundle in [Visual-Generative-Foundation-Model-Collection](https://github.com/Bili-Sakura/Visual-Generative-Foundation-Model-Collection). Inference only needs:
- This model repo (`BiliSakura/FiT-diffusers`)
- PyPI `diffusers`, `torch`, `safetensors`
Each variant subfolder is a self-contained Diffusers model repo with:
- `model_index.json` (includes ImageNet `id2label`)
- `pipeline.py` (`FiTPipeline` for FiTv1, `FiTv2Pipeline` for FiTv2)
- `transformer/fit_transformer_2d.py` and weights
- `scheduler/scheduler_config.json`
- `vae/diffusion_pytorch_model.safetensors` (`stabilityai/sd-vae-ft-ema`)
## Demo
Class-conditional sample (ImageNet class **207**, golden retriever), `FiTv1-XL/2` at 256×256, 250 steps, CFG 1.5, seed 42.
## Available checkpoints
| Checkpoint | Path | Resolution | Sampler | Steps | CFG | FID (native res) |
| --- | --- | --- | --- | --- | --- | --- |
| FiTv1-XL/2 | [`FiTv1-XL-2-256/`](FiTv1-XL-2-256/) | 256×256 | improved diffusion (DDPM respaced) | 250 | 1.5 | 4.21 |
| FiTv2-XL/2 | [`FiTv2-XL-2-256/`](FiTv2-XL-2-256/) | 256×256 | flow matching (velocity ODE) | 250 | 1.5 | 2.26 |
| FiTv2-3B/2 | [`FiTv2-3B-2-256/`](FiTv2-3B-2-256/) | 256×256 | flow matching (velocity ODE) | 250 | 1.5 | 2.15 |
| FiTv2-HR-XL/2 | [`FiTv2-XL-2-512/`](FiTv2-XL-2-512/) | 512×512 | flow matching (velocity ODE) | 250 | 1.5 | 2.90 |
| FiTv2-HR-3B/2 | [`FiTv2-3B-2-512/`](FiTv2-3B-2-512/) | 512×512 | flow matching (velocity ODE) | 250 | 1.5 | 2.41 |
## Inference
```python
from pathlib import Path
import torch
from diffusers import DiffusionPipeline
model_dir = Path("./FiTv2-XL-2-256")
pipe = DiffusionPipeline.from_pretrained(
str(model_dir),
local_files_only=True,
custom_pipeline=str(model_dir / "pipeline.py"),
trust_remote_code=True,
torch_dtype=torch.bfloat16,
).to("cuda")
generator = torch.Generator(device="cuda").manual_seed(42)
image = pipe(
class_labels="golden retriever",
height=256,
width=256,
num_inference_steps=250,
guidance_scale=1.5,
generator=generator,
).images[0]
image.save("demo.png")
```
Load a **variant subfolder** (e.g. `./FiTv2-XL-2-256`), not the repo root. For FiTv1, use `./FiTv1-XL-2-256` with the same call pattern (`FiTPipeline` + DDPM scheduler).
## Repo layout
```text
BiliSakura/FiT-diffusers/
├── README.md
├── FiTv1-XL-2-256/
├── FiTv2-XL-2-256/
├── FiTv2-3B-2-256/
├── FiTv2-XL-2-512/
└── FiTv2-3B-2-512/
├── README.md
├── model_index.json
├── pipeline.py
├── demo.png
├── transformer/
│ ├── config.json
│ ├── fit_transformer_2d.py
│ └── diffusion_pytorch_model.safetensors
├── vae/
│ ├── config.json
│ └── diffusion_pytorch_model.safetensors
└── scheduler/
└── scheduler_config.json
```
|