--- license: cc-by-nc-4.0 library_name: pytorch tags: - image-generation - diffusion - imagenet - representation-autoencoder --- # FuseLift: DiT-XL on the lifted decoder state h0 Class-conditional ImageNet-256 DiT-XL checkpoints that diffuse **h0**, the input state of a frozen representation-autoencoder pixel decoder, instead of the fused encoder latent `z`. ```text image -> DINOv3-L -> layer fusion + final-layer token mean -> z [1024, 16, 16] -> decoder_embed(z) + position embedding -> h0 [1152, 16, 16] <- DiT target h0 (sampled) -> all 28 decoder blocks -> pixels ``` Code, configs and instructions: [github.com/Hongyang-Du/FuseLift](https://github.com/Hongyang-Du/FuseLift). ## Files | File | Content | | --- | --- | | `k7/ditxl_h0_raev2k7_ep080.safetensors` | FuseLift K=7 DiT-XL (`DiTwDDTHeadIG`, 876.2M params), EMA weights only, epoch 80, step 100,080 | | `k7/h0_stats_raev2k7.pt` | per-position `mean` / `var` of h0 (`[1152, 16, 16]`) and `mean_cls` (`[1, 1, 1152]`), 50k ImageNet train images | | `k7/decoder_raev2k7.pt` | official RAEv2 DINOv3-L K=7 decoder (layers 11, 13, ..., 23), unchanged | | `k23/ditxl_h0_raev2k23_ep080.safetensors` | FuseLift K=23 DiT-XL, EMA weights only, epoch 80, step 100,080 | | `k23/h0_stats_raev2k23.pt` | h0 statistics for the K=23 decoder, same format | | `k23/decoder_raev2k23.pt` | official RAEv2 DINOv3-L K=23 decoder (layers 1, 2, ..., 23), unchanged | | `samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz` | the 50,000 guided K=7 samples behind the 0.98 gFID: `arr_0`, uint8 `[50000, 256, 256, 3]`, 50 per class | | `reference/imagenet_256_fid_stats.npz` | the FID reference behind every gFID here: Inception `mu` (`[2048]`) and `sigma` (`[2048, 2048]`) of ImageNet-256 train images. Unchanged copy of the default fd_evaluator reference from [nanovisionx/nanogen-evals-stats](https://huggingface.co/datasets/nanovisionx/nanogen-evals-stats) (SHA-256 `925e8b5b…0775f1ac`) | The two decoders are byte-identical copies of `stage1/imagenet/dinov3l-k7/decoder.pt` and `stage1/imagenet/dinov3l-k23/decoder.pt` from [nyu-visionx/RAEv2-models](https://huggingface.co/nyu-visionx/RAEv2-models). They are frozen during FuseLift training and are included so that each generator ships with the decoder that reads it. Credit for them goes to the RAEv2 authors. ## Results ImageNet-256, 50,000 samples, 50 Euler steps, EMA weights. FID is computed against ImageNet **train** statistics (the official RAEv2 reference). IG = internal guidance, scale 1.78 on t in [0.1, 1]. FDr6 is the mean Fréchet distance over six feature extractors. Each cell is gFID / IS / FDr6. Epoch 80, two sampling seeds: | Model | Seed | No guidance | IG 1.78 | | --- | --- | --- | --- | | FuseLift K=7 | 1 | 1.77 / 208.8 / 4.00 | **0.98** / 253.4 / 2.14 | | FuseLift K=7 | 42 | 1.76 / 209.2 / 3.97 | 1.01 / 253.4 / 2.13 | | FuseLift K=23 | 1 | 2.96 / 195.4 / 5.77 | 1.07 / 256.1 / 2.30 | | FuseLift K=23 | 42 | 2.95 / 193.6 / 5.77 | 1.08 / 254.7 / 2.30 | gFID during training (sampling seed 42, no guidance / IG 1.78): | Model | Epoch 20 | Epoch 40 | Epoch 60 | Epoch 80 | | --- | --- | --- | --- | --- | | FuseLift K=7 | 2.44 / 2.11 | 1.88 / 1.32 | 1.78 / 1.02 | 1.76 / 1.01 | | FuseLift K=23 | 3.24 / 2.10 | 2.76 / 1.34 | 2.97 / 1.08 | 2.95 / 1.08 | Same K=7 decoder and protocol, generator on the encoder latent `z` (sampling seed 42): | Generator (K=7, DiT-XL, 80 epochs) | No guidance | IG 1.78 | | --- | --- | --- | | official RAEv2 checkpoint, on `z` | 1.65 / 225.5 / 3.92 | 1.08 / 259.1 / 2.08 | | our retraining on `z`, FuseLift schedule | 1.65 / 226.7 / 4.11 | 1.13 / 260.0 / 2.19 | | FuseLift, on h0 | 1.76 / 209.2 / 3.97 | 1.01 / 253.4 / 2.13 | Each number is one 50k evaluation of one training run. ## Use ```bash git clone https://github.com/Hongyang-Du/FuseLift.git && cd FuseLift python -m pip install -r requirements.txt hf download Hongyang-Du/FuseLift --local-dir checkpoints/fuselift python src/sample_dit.py \ --config configs/stage2/training/fuselift-h0-raev2k7-ditxl.yaml \ --ckpt checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors \ --classes 1 207 388 980 --per-class 4 --out outputs/fuselift_k7_grid.png ``` Check the reported guided K=7 numbers from the released samples: ```bash pip install --no-deps "git+https://github.com/nanovisionx/nanogen-evals.git@19dfb4c2705333eb8b97e454fb354d47d1fe135b#subdirectory=fd_evaluator" python scripts/eval_npz.py checkpoints/fuselift/samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz # fid 0.98, inception_score 253.4, fdr6 2.14 ``` Loading a generator by hand: ```python from safetensors.torch import load_file state = load_file("checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors") # EMA DiT-XL model.load_state_dict(state) # stage2.models.DDT.DiTwDDTHeadIG, in_channels=1152 ``` The checkpoints were trained with internal training code; the GitHub README describes how the public code was checked against it. The DINOv3 encoder is not included and is obtained from its owner under its own license.