FuseLift / README.md
Hongyang-Du's picture
Add the FID reference statistics
cbf344e verified
|
Raw History Blame Contribute Delete
5.08 kB
metadata
license: cc-by-nc-4.0
library_name: pytorch
tags:
  - image-generation
  - diffusion
  - imagenet
  - representation-autoencoder

FuseLift: DiT-XL on the lifted decoder state h0

Class-conditional ImageNet-256 DiT-XL checkpoints that diffuse h0, the input state of a frozen representation-autoencoder pixel decoder, instead of the fused encoder latent z.

image -> DINOv3-L -> layer fusion + final-layer token mean          -> z   [1024, 16, 16]
      -> decoder_embed(z) + position embedding                      -> h0  [1152, 16, 16]   <- DiT target
h0 (sampled) -> all 28 decoder blocks -> pixels

Code, configs and instructions: github.com/Hongyang-Du/FuseLift.

Files

File Content
k7/ditxl_h0_raev2k7_ep080.safetensors FuseLift K=7 DiT-XL (DiTwDDTHeadIG, 876.2M params), EMA weights only, epoch 80, step 100,080
k7/h0_stats_raev2k7.pt per-position mean / var of h0 ([1152, 16, 16]) and mean_cls ([1, 1, 1152]), 50k ImageNet train images
k7/decoder_raev2k7.pt official RAEv2 DINOv3-L K=7 decoder (layers 11, 13, ..., 23), unchanged
k23/ditxl_h0_raev2k23_ep080.safetensors FuseLift K=23 DiT-XL, EMA weights only, epoch 80, step 100,080
k23/h0_stats_raev2k23.pt h0 statistics for the K=23 decoder, same format
k23/decoder_raev2k23.pt official RAEv2 DINOv3-L K=23 decoder (layers 1, 2, ..., 23), unchanged
samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz the 50,000 guided K=7 samples behind the 0.98 gFID: arr_0, uint8 [50000, 256, 256, 3], 50 per class
reference/imagenet_256_fid_stats.npz the FID reference behind every gFID here: Inception mu ([2048]) and sigma ([2048, 2048]) of ImageNet-256 train images. Unchanged copy of the default fd_evaluator reference from nanovisionx/nanogen-evals-stats (SHA-256 925e8b5b…0775f1ac)

The two decoders are byte-identical copies of stage1/imagenet/dinov3l-k7/decoder.pt and stage1/imagenet/dinov3l-k23/decoder.pt from nyu-visionx/RAEv2-models. They are frozen during FuseLift training and are included so that each generator ships with the decoder that reads it. Credit for them goes to the RAEv2 authors.

Results

ImageNet-256, 50,000 samples, 50 Euler steps, EMA weights. FID is computed against ImageNet train statistics (the official RAEv2 reference). IG = internal guidance, scale 1.78 on t in [0.1, 1]. FDr6 is the mean Fréchet distance over six feature extractors. Each cell is gFID / IS / FDr6.

Epoch 80, two sampling seeds:

Model Seed No guidance IG 1.78
FuseLift K=7 1 1.77 / 208.8 / 4.00 0.98 / 253.4 / 2.14
FuseLift K=7 42 1.76 / 209.2 / 3.97 1.01 / 253.4 / 2.13
FuseLift K=23 1 2.96 / 195.4 / 5.77 1.07 / 256.1 / 2.30
FuseLift K=23 42 2.95 / 193.6 / 5.77 1.08 / 254.7 / 2.30

gFID during training (sampling seed 42, no guidance / IG 1.78):

Model Epoch 20 Epoch 40 Epoch 60 Epoch 80
FuseLift K=7 2.44 / 2.11 1.88 / 1.32 1.78 / 1.02 1.76 / 1.01
FuseLift K=23 3.24 / 2.10 2.76 / 1.34 2.97 / 1.08 2.95 / 1.08

Same K=7 decoder and protocol, generator on the encoder latent z (sampling seed 42):

Generator (K=7, DiT-XL, 80 epochs) No guidance IG 1.78
official RAEv2 checkpoint, on z 1.65 / 225.5 / 3.92 1.08 / 259.1 / 2.08
our retraining on z, FuseLift schedule 1.65 / 226.7 / 4.11 1.13 / 260.0 / 2.19
FuseLift, on h0 1.76 / 209.2 / 3.97 1.01 / 253.4 / 2.13

Each number is one 50k evaluation of one training run.

Use

git clone https://github.com/Hongyang-Du/FuseLift.git && cd FuseLift
python -m pip install -r requirements.txt
hf download Hongyang-Du/FuseLift --local-dir checkpoints/fuselift

python src/sample_dit.py \
  --config configs/stage2/training/fuselift-h0-raev2k7-ditxl.yaml \
  --ckpt checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors \
  --classes 1 207 388 980 --per-class 4 --out outputs/fuselift_k7_grid.png

Check the reported guided K=7 numbers from the released samples:

pip install --no-deps "git+https://github.com/nanovisionx/nanogen-evals.git@19dfb4c2705333eb8b97e454fb354d47d1fe135b#subdirectory=fd_evaluator"
python scripts/eval_npz.py checkpoints/fuselift/samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz
# fid 0.98, inception_score 253.4, fdr6 2.14

Loading a generator by hand:

from safetensors.torch import load_file
state = load_file("checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors")   # EMA DiT-XL
model.load_state_dict(state)                # stage2.models.DDT.DiTwDDTHeadIG, in_channels=1152

The checkpoints were trained with internal training code; the GitHub README describes how the public code was checked against it. The DINOv3 encoder is not included and is obtained from its owner under its own license.