FuseLift: DiT-XL on the lifted decoder state h0
Class-conditional ImageNet-256 DiT-XL checkpoints that diffuse h0, the input state of a frozen
representation-autoencoder pixel decoder, instead of the fused encoder latent z.
image -> DINOv3-L -> layer fusion + final-layer token mean -> z [1024, 16, 16]
-> decoder_embed(z) + position embedding -> h0 [1152, 16, 16] <- DiT target
h0 (sampled) -> all 28 decoder blocks -> pixels
Code, configs and instructions: github.com/Hongyang-Du/FuseLift.
Files
| File | Content |
|---|---|
k7/ditxl_h0_raev2k7_ep080.safetensors |
FuseLift K=7 DiT-XL (DiTwDDTHeadIG, 876.2M params), EMA weights only, epoch 80, step 100,080 |
k7/h0_stats_raev2k7.pt |
per-position mean / var of h0 ([1152, 16, 16]) and mean_cls ([1, 1, 1152]), 50k ImageNet train images |
k7/decoder_raev2k7.pt |
official RAEv2 DINOv3-L K=7 decoder (layers 11, 13, ..., 23), unchanged |
k23/ditxl_h0_raev2k23_ep080.safetensors |
FuseLift K=23 DiT-XL, EMA weights only, epoch 80, step 100,080 |
k23/h0_stats_raev2k23.pt |
h0 statistics for the K=23 decoder, same format |
k23/decoder_raev2k23.pt |
official RAEv2 DINOv3-L K=23 decoder (layers 1, 2, ..., 23), unchanged |
samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz |
the 50,000 guided K=7 samples behind the 0.98 gFID: arr_0, uint8 [50000, 256, 256, 3], 50 per class |
reference/imagenet_256_fid_stats.npz |
the FID reference behind every gFID here: Inception mu ([2048]) and sigma ([2048, 2048]) of ImageNet-256 train images. Unchanged copy of the default fd_evaluator reference from nanovisionx/nanogen-evals-stats (SHA-256 925e8b5b…0775f1ac) |
The two decoders are byte-identical copies of stage1/imagenet/dinov3l-k7/decoder.pt and
stage1/imagenet/dinov3l-k23/decoder.pt from
nyu-visionx/RAEv2-models. They are frozen during
FuseLift training and are included so that each generator ships with the decoder that reads it.
Credit for them goes to the RAEv2 authors.
Results
ImageNet-256, 50,000 samples, 50 Euler steps, EMA weights. FID is computed against ImageNet train statistics (the official RAEv2 reference). IG = internal guidance, scale 1.78 on t in [0.1, 1]. FDr6 is the mean Fréchet distance over six feature extractors. Each cell is gFID / IS / FDr6.
Epoch 80, two sampling seeds:
| Model | Seed | No guidance | IG 1.78 |
|---|---|---|---|
| FuseLift K=7 | 1 | 1.77 / 208.8 / 4.00 | 0.98 / 253.4 / 2.14 |
| FuseLift K=7 | 42 | 1.76 / 209.2 / 3.97 | 1.01 / 253.4 / 2.13 |
| FuseLift K=23 | 1 | 2.96 / 195.4 / 5.77 | 1.07 / 256.1 / 2.30 |
| FuseLift K=23 | 42 | 2.95 / 193.6 / 5.77 | 1.08 / 254.7 / 2.30 |
gFID during training (sampling seed 42, no guidance / IG 1.78):
| Model | Epoch 20 | Epoch 40 | Epoch 60 | Epoch 80 |
|---|---|---|---|---|
| FuseLift K=7 | 2.44 / 2.11 | 1.88 / 1.32 | 1.78 / 1.02 | 1.76 / 1.01 |
| FuseLift K=23 | 3.24 / 2.10 | 2.76 / 1.34 | 2.97 / 1.08 | 2.95 / 1.08 |
Same K=7 decoder and protocol, generator on the encoder latent z (sampling seed 42):
| Generator (K=7, DiT-XL, 80 epochs) | No guidance | IG 1.78 |
|---|---|---|
official RAEv2 checkpoint, on z |
1.65 / 225.5 / 3.92 | 1.08 / 259.1 / 2.08 |
our retraining on z, FuseLift schedule |
1.65 / 226.7 / 4.11 | 1.13 / 260.0 / 2.19 |
| FuseLift, on h0 | 1.76 / 209.2 / 3.97 | 1.01 / 253.4 / 2.13 |
Each number is one 50k evaluation of one training run.
Use
git clone https://github.com/Hongyang-Du/FuseLift.git && cd FuseLift
python -m pip install -r requirements.txt
hf download Hongyang-Du/FuseLift --local-dir checkpoints/fuselift
python src/sample_dit.py \
--config configs/stage2/training/fuselift-h0-raev2k7-ditxl.yaml \
--ckpt checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors \
--classes 1 207 388 980 --per-class 4 --out outputs/fuselift_k7_grid.png
Check the reported guided K=7 numbers from the released samples:
pip install --no-deps "git+https://github.com/nanovisionx/nanogen-evals.git@19dfb4c2705333eb8b97e454fb354d47d1fe135b#subdirectory=fd_evaluator"
python scripts/eval_npz.py checkpoints/fuselift/samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz
# fid 0.98, inception_score 253.4, fdr6 2.14
Loading a generator by hand:
from safetensors.torch import load_file
state = load_file("checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors") # EMA DiT-XL
model.load_state_dict(state) # stage2.models.DDT.DiTwDDTHeadIG, in_channels=1152
The checkpoints were trained with internal training code; the GitHub README describes how the public code was checked against it. The DINOv3 encoder is not included and is obtained from its owner under its own license.