FuseLift: DiT-XL on the lifted decoder state h0

Class-conditional ImageNet-256 DiT-XL checkpoints that diffuse h0, the input state of a frozen representation-autoencoder pixel decoder, instead of the fused encoder latent z.

image -> DINOv3-L -> layer fusion + final-layer token mean          -> z   [1024, 16, 16]
      -> decoder_embed(z) + position embedding                      -> h0  [1152, 16, 16]   <- DiT target
h0 (sampled) -> all 28 decoder blocks -> pixels

Code, configs and instructions: github.com/Hongyang-Du/FuseLift.

Files

File Content
k7/ditxl_h0_raev2k7_ep080.safetensors FuseLift K=7 DiT-XL (DiTwDDTHeadIG, 876.2M params), EMA weights only, epoch 80, step 100,080
k7/h0_stats_raev2k7.pt per-position mean / var of h0 ([1152, 16, 16]) and mean_cls ([1, 1, 1152]), 50k ImageNet train images
k7/decoder_raev2k7.pt official RAEv2 DINOv3-L K=7 decoder (layers 11, 13, ..., 23), unchanged
k23/ditxl_h0_raev2k23_ep080.safetensors FuseLift K=23 DiT-XL, EMA weights only, epoch 80, step 100,080
k23/h0_stats_raev2k23.pt h0 statistics for the K=23 decoder, same format
k23/decoder_raev2k23.pt official RAEv2 DINOv3-L K=23 decoder (layers 1, 2, ..., 23), unchanged
samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz the 50,000 guided K=7 samples behind the 0.98 gFID: arr_0, uint8 [50000, 256, 256, 3], 50 per class
reference/imagenet_256_fid_stats.npz the FID reference behind every gFID here: Inception mu ([2048]) and sigma ([2048, 2048]) of ImageNet-256 train images. Unchanged copy of the default fd_evaluator reference from nanovisionx/nanogen-evals-stats (SHA-256 925e8b5b…0775f1ac)

The two decoders are byte-identical copies of stage1/imagenet/dinov3l-k7/decoder.pt and stage1/imagenet/dinov3l-k23/decoder.pt from nyu-visionx/RAEv2-models. They are frozen during FuseLift training and are included so that each generator ships with the decoder that reads it. Credit for them goes to the RAEv2 authors.

Results

ImageNet-256, 50,000 samples, 50 Euler steps, EMA weights. FID is computed against ImageNet train statistics (the official RAEv2 reference). IG = internal guidance, scale 1.78 on t in [0.1, 1]. FDr6 is the mean Fréchet distance over six feature extractors. Each cell is gFID / IS / FDr6.

Epoch 80, two sampling seeds:

Model Seed No guidance IG 1.78
FuseLift K=7 1 1.77 / 208.8 / 4.00 0.98 / 253.4 / 2.14
FuseLift K=7 42 1.76 / 209.2 / 3.97 1.01 / 253.4 / 2.13
FuseLift K=23 1 2.96 / 195.4 / 5.77 1.07 / 256.1 / 2.30
FuseLift K=23 42 2.95 / 193.6 / 5.77 1.08 / 254.7 / 2.30

gFID during training (sampling seed 42, no guidance / IG 1.78):

Model Epoch 20 Epoch 40 Epoch 60 Epoch 80
FuseLift K=7 2.44 / 2.11 1.88 / 1.32 1.78 / 1.02 1.76 / 1.01
FuseLift K=23 3.24 / 2.10 2.76 / 1.34 2.97 / 1.08 2.95 / 1.08

Same K=7 decoder and protocol, generator on the encoder latent z (sampling seed 42):

Generator (K=7, DiT-XL, 80 epochs) No guidance IG 1.78
official RAEv2 checkpoint, on z 1.65 / 225.5 / 3.92 1.08 / 259.1 / 2.08
our retraining on z, FuseLift schedule 1.65 / 226.7 / 4.11 1.13 / 260.0 / 2.19
FuseLift, on h0 1.76 / 209.2 / 3.97 1.01 / 253.4 / 2.13

Each number is one 50k evaluation of one training run.

Use

git clone https://github.com/Hongyang-Du/FuseLift.git && cd FuseLift
python -m pip install -r requirements.txt
hf download Hongyang-Du/FuseLift --local-dir checkpoints/fuselift

python src/sample_dit.py \
  --config configs/stage2/training/fuselift-h0-raev2k7-ditxl.yaml \
  --ckpt checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors \
  --classes 1 207 388 980 --per-class 4 --out outputs/fuselift_k7_grid.png

Check the reported guided K=7 numbers from the released samples:

pip install --no-deps "git+https://github.com/nanovisionx/nanogen-evals.git@19dfb4c2705333eb8b97e454fb354d47d1fe135b#subdirectory=fd_evaluator"
python scripts/eval_npz.py checkpoints/fuselift/samples/ditxl_h0_raev2k7_ep080_ig178_50k.npz
# fid 0.98, inception_score 253.4, fdr6 2.14

Loading a generator by hand:

from safetensors.torch import load_file
state = load_file("checkpoints/fuselift/k7/ditxl_h0_raev2k7_ep080.safetensors")   # EMA DiT-XL
model.load_state_dict(state)                # stage2.models.DDT.DiTwDDTHeadIG, in_channels=1152

The checkpoints were trained with internal training code; the GitHub README describes how the public code was checked against it. The DINOv3 encoder is not included and is obtained from its owner under its own license.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support