Blame for README.md

---
license: cc-by-nc-4.0
library_name: pytorch
pipeline_tag: text-to-image
tags:
  - diffusion
  - latent-diffusion
  - end-to-end-training
  - variational-autoencoder
  - text-to-image
  - class-conditional-image-generation
  - sit
  - mmdit
---

# GenFirst

Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling.

- Paper: [arXiv:2608.29335](https://arxiv.org/abs/2608.29335)
- Code: [github.com/ByteDance-Seed/GenFirst](https://github.com/ByteDance-Seed/GenFirst)
- Project page: [tom-zgt.github.io/GenFirst](https://tom-zgt.github.io/GenFirst/)

These weights are released under **CC BY-NC 4.0: non-commercial research use
only**, which is stricter than the MIT licence on the code. See
[Licence and third-party terms](#licence-and-third-party-terms) before use.

## What this is

GenFirst trains a latent VAE jointly with a generative prior without the latent
collapse that usually makes such end-to-end training unstable. The method is a
two-phase schedule over the prior and entropy weights: phase 1 lets the prior
shape the latent geometry, phase 2 drops both weights tenfold so the decoder
recovers detail. The result is a latent space that a diffusion prior fits well
while remaining a good autoencoder.

This release covers the **EiT** (diffusion-prior) instantiation. The
autoregressive **EAR** instantiation from the paper is not released.

Two kinds of artifact are published, and they are used together:

1. **End-to-end VAEs** — the encoder/decoder produced by the end-to-end stage,
   each shipped with its latent normalization statistics.
2. **Generative priors** — SiT models for class-conditional generation, and
   MMDiT models for text-to-image, each trained on a frozen VAE from (1).

## Model details

| | |
|---|---|
| Developed by | Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang |
| Model type | Latent diffusion (flow matching / stochastic interpolant) with a jointly trained VAE |
| Prior architecture | SiT-XL and MMDiT-XL: 28 blocks, width 1152, 16 attention heads (~675M parameters) |
| Text encoder (text-to-image) | CLIP-L, frozen, not modified by us |
| Precision | Weights released in fp32; sampling tested in bf16 |
| Framework | PyTorch |
| Inference code | Required — see [Usage](#usage). These are raw `.pt` checkpoints, not `diffusers` pipelines. |

### Class-conditional priors

`SiT-XL` on a frozen end-to-end VAE. Conditioning is a single integer class
label in `[0, 999]` following the standard 1000-class label ordering; there is
no text input. Epoch counts are prior-only training epochs.

| Checkpoint | Resolution | VAE | gFID | IS | gFID (guided) | IS (guided) | Download |
|---|---|---|---|---|---|---|---|
| EiT, 80 epochs | 256x256 | `f16d32` | 2.79 | 175.4 | 1.38 | 271.3 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-80ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) |
| EiT, 480 epochs | 256x256 | `f16d32` | 1.60 | 216.8 | 0.988 | 270.7 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-480ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) |
| EiT, 800 epochs | 256x256 | `f16d32` | **1.45** | 226.7 | **0.969** | 283.4 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-800ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) |
| EiT, 80 epochs | 512x512 | `f16d32` | – | – | 1.63 | 278.9 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet512/eit-imagenet512-80ep.pt) |
| EiT, 400 epochs | 512x512 | `f16d32` | – | – | **1.23** | 286.1 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet512/eit-imagenet512-400ep.pt) |

The 512 models reuse the 256 `f16d32` VAE unchanged and frozen; only the prior
is trained at 512.

### Text-to-image priors

`MMDiT-XL/2` with a frozen CLIP-L text encoder, on a frozen end-to-end FLUX
VAE, at 512x512. Scores are overall GenEval and DPG-Bench, with no prompt
rewriting. The two variants differ only in which VAE they were trained on.

| Checkpoint | Resolution | GenEval | DPG-Bench | Download |
|---|---|---|---|---|
| EiT T2I, variant A | 512x512 | **0.8987** | **82.60** | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/text-to-image/variant-a/eit-t2i-variant-a-512.pt) · [VAE](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-sit-imagenet-s2) |
| EiT T2I, variant B | 512x512 | 0.8920 | 82.06 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/text-to-image/variant-b/eit-t2i-variant-b-512.pt) · [VAE](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-mmdit-l2-t2i-s2) |

### Shared visual latents (generation + understanding)

`MMDiT-XL/2` on a frozen Qwen3-VL ViT autoencoder (`--vae-arch qwenvit`) shaped end-to-end with a generative prior plus representation supervision (`SigLIP` or `VLM-NLL`, Table 9 in the paper).

| Checkpoint | VAE bundle | Supervision | Resolution | rFID ↓ | PSNR ↑ | GenEval ↑ | DPG-Bench ↑ | Linear Probe Top-1 (%) ↑ |
|---|---|---|---|---|---|---|---|---|
| [`shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt) | [`vae/qwenvit-siglip-b256-s2/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-siglip-b256-s2) | E2E + SigLIP | 512x512 | **0.39** | **27.64** | **0.90** | **83.76** | 79.25 |
| [`shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt) | [`vae/qwenvit-nlltrunk-s2/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-nlltrunk-s2) | E2E + VLM-NLL | 512x512 | 0.46 | 27.19 | 0.89 | 83.49 | **81.69** |

### Unified text-image modeling (`JointMMDiT-XL/2`)

Bidirectional block-causal `JointMMDiT-XL/2` trained on the jointly E2E-shaped text VAE (Cola) and image VAE (FLUX.1) (`vae/e2e-joint-gan-l2-330k/`, Table 10 in the paper). The phase-3 SFT checkpoint (`-512-sft.pt`) is used for text-to-image (GenEval / DPG-Bench), and the phase-2 pre-SFT checkpoint (`-512.pt`) is used for image-to-text COCO captioning.

| Checkpoints | Setting | GenEval ↑ | DPG-Bench ↑ | CIDEr ↑ | BLEU-4 ↑ | METEOR ↑ | ROUGE-L ↑ | SPICE ↑ | CLIPScore ↑ |
|---|---|---|---|---|---|---|---|---|---|
| [`eit-joint-e2e-text-e2e-img-512-sft.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt) (T2I) & [`eit-joint-e2e-text-e2e-img-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512.pt) (I2T) | E2E-joint text + image VAEs ([`vae/e2e-joint-gan-l2-330k/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-joint-gan-l2-330k)) | **86.57** | **72.68** | **33.28** | **7.76** | **17.74** | **34.16** | **11.81** | **57.97** |

### End-to-end VAEs

Reconstruction quality of the posterior mean, measured at the resolution the
VAE is used at.

| VAE | Downsample / channels | Initialized from | rFID | PSNR | SSIM | LPIPS | Download |
|---|---|---|---|---|---|---|---|
| `f8d4` | 8x, 4ch | SD-VAE (`stabilityai/sd-vae-ft-ema`) | **0.57** | 25.13 | 0.71 | 0.14 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f8d4) |
| `f16d32` (decoder-ft, @256) | 16x, 32ch | VA-VAE (LightningDiT) | 0.36 | 25.82 | 0.74 | 0.12 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f16d32) |
| `f16d32` (no dec-ft, @512) | 16x, 32ch | VA-VAE (LightningDiT) | – | – | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f16d32-nodecft) |
| `f8d16` variant A | 8x, 16ch | FLUX.1 VAE | 0.23 | 29.43 | 0.85 | 0.07 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-sit-imagenet-s2) |
| `f8d16` variant B | 8x, 16ch | FLUX.1 VAE | 0.21 | 29.37 | 0.85 | 0.07 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-mmdit-l2-t2i-s2) |
| `qwenvit-siglip-b256-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (SigLIP) | 0.39 | 27.64 | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-siglip-b256-s2) |
| `qwenvit-nlltrunk-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (VLM-NLL) | 0.46 | 27.19 | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-nlltrunk-s2) |
| `e2e-joint-gan-l2-330k` | 8x, 16ch (image) + 16d (text) | FLUX.1 VAE + Cola Text VAE | – | – | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-joint-gan-l2-330k) |
| `f8d32` (`e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, @512) | 8x, 32ch | FLUX.2 VAE | **0.041** | **36.46** | **0.953** | **0.026** | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k) |

**A VAE and its latent statistics must be kept together.** The prior was
trained on latents normalized by those exact statistics; pairing a VAE with the
wrong statistics file produces noise, not a degraded image. Each VAE directory
is laid out as:

```
<name>/
  <name>.pt                     VAE weights
  <name>.safetensors            VAE weights in diffusers safetensors format (for FLUX.2 release)
  <name>-latents-stats.pt       latent normalization (scale + bias)
  <name>-discriminator-ckpt.pt  discriminator, only needed to resume end-to-end training
  config.json / vae_params.json diffusers config and explicit scaling_factor / shift_factor (for FLUX.2 release)
```

For `vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, the per-channel `scaling_factor` and `shift_factor` parameters are:

```bash
--vae_pth=vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors
--model.vae.scaling_factor=[0.887221,0.886926,0.887096,0.881914,0.886901,0.887307,0.886685,0.886833,0.886540,0.886646,0.887031,0.887073,0.887381,0.886768,0.887299,0.885674,0.886466,0.886795,0.886979,0.886519,0.886949,0.886949,0.887036,0.886828,0.887344,0.887004,0.886816,0.886799,0.887354,0.887523,0.887012,0.886643]
--model.vae.shift_factor=[-0.000623,0.000143,0.000282,0.001599,-0.001093,-0.000477,0.000855,-0.000830,-0.000883,0.001076,-0.000417,0.000200,0.000626,-0.000649,-0.000137,0.001453,-0.000816,-0.000307,-0.000290,0.000662,0.000484,0.000993,0.000574,-0.000139,-0.000914,0.000578,-0.000691,-0.000323,0.000619,-0.000606,-0.000372,0.000437]
```

## Usage

These checkpoints need the training repository; there is no `diffusers`
integration and no `from_pretrained` entry point.

```bash
git clone https://github.com/ByteDance-Seed/GenFirst
cd GenFirst
pip install -r requirements.txt          # install torch first
```

Class-conditional sampling, writing a small batch of images rather than a full
evaluation run:

```bash
cd Prior_only
python generate.py \
    --model SiT-XL/1 --resolution 256 \
    --ckpt /path/to/prior.pt \
    --vae-arch f16d32 \
    --vae-ckpt /path/to/e2e-f16d32.pt \
    --latents-stats /path/to/e2e-f16d32-latents-stats.pt \
    --num-fid-samples 64 --label-sampling random \
    --cfg-scale 1.5 --sample-dir samples
```

`--model` must match the VAE: `SiT-XL/2` for `f8d4`, `SiT-XL/1` for `f16d32`.
Text-to-image sampling, the guidance settings that reproduce the numbers above,
and the full evaluation harness are documented in the repository under `docs/`.

## Evaluation

Reported metrics and their protocols:

- **gFID / IS** — 50,000 generated samples scored with
  [torch-fidelity](https://github.com/LTH14/torch-fidelity) against the
  reference statistics used by [MAR](https://github.com/LTH14/mar), at the
  stated resolution. Sampling is SDE with 250 steps. Guided rows use
  [autoguidance](https://arxiv.org/abs/2406.02507) (pointing away from an
  early checkpoint of the same run) with CFG on top, except the 512 rows, which
  use plain CFG.
- **rFID / PSNR / SSIM / LPIPS** — posterior-mean reconstructions over a
  50,000-image validation split, following
  [latent-diffusion](https://github.com/CompVis/latent-diffusion). rFID here is
  measured against those validation images, not a training-set reference, so it
  is not comparable to numbers computed the other way.
- **GenEval / DPG-Bench** — overall scores, Euler sampling with 50 steps and
  CFG 7.0 on the interval `[0.35, 1.0]`, no prompt rewriting.

Numbers were produced with the versions pinned in `environment.yml`. FID is
sensitive to reference statistics and to the resizing path, so comparisons
against other work are only meaningful under the same protocol.

## Training

Training proceeds in two stages. The end-to-end stage trains the VAE jointly
with a prior under the two-phase GenFirst schedule (global batch 256; 400k
steps at the high weight, then 200k at the low one). The prior-only stage then
freezes the VAE, caches its latents, and trains from scratch the prior that is
actually evaluated (4M steps at 256, 2M at 512). An optional 10k-step
decoder-only finetune between the stages improves unguided gFID by roughly 0.2.
Text-to-image follows the same structure, with an `MMDiT-L/2` prior shaping the
latent during the end-to-end stage and the released `MMDiT-XL/2` trained on the
frozen result.

Training data consists of publicly available datasets. Individual corpora are
not enumerated here; each is subject to its own licence and terms of use, which
govern independently of the licence on these weights.

## Intended use

Intended for research: reproducing the paper, studying end-to-end VAE and prior
training, latent-space analysis, and use as a baseline or initialization for
further non-commercial research.

**Out of scope.** Any commercial use. Generating content about real,
identifiable people. Generating sexual, violent, hateful, or otherwise harmful
imagery. Any use presenting outputs as real photographs or as factual records.
Any safety-critical or decision-making pipeline. These models produce synthetic
images with no factual grounding and must not be treated as a source of truth.

No safety filter, watermarking, or content classifier is included. Anyone
deploying these weights in a setting where outputs reach other people is
responsible for adding those controls.

## Limitations and biases

- The class-conditional models generate from a fixed 1000-class label set and
  cannot follow text. Their outputs reflect the visual distribution and the
  category taxonomy they were fit to, including the geographic and cultural
  skew documented for that style of curated image collection.
- The text-to-image models inherit the biases of both their training data and
  the frozen CLIP-L text encoder. Expect uneven representation across
  demographics, professions, and cultures, and expect prompt-following to
  degrade on long, compositional, or counting-heavy prompts. GenEval and
  DPG-Bench scores measure prompt alignment on those benchmarks only and do not
  certify safety or fairness.
- Text rendering, hands, and fine structure remain unreliable, as with
  comparable models at this scale.
- Metrics reported here are distribution-level. A good FID says nothing about
  the quality or appropriateness of any single sample.
- The 512 class-conditional models are reported with guidance only; their
  unguided behaviour is untuned.
- A VAE paired with mismatched latent statistics fails silently in the sense
  that it produces output rather than an error. Check the pairing.

## Licence and third-party terms

**Weights: [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) —
non-commercial research use only, with attribution.** This is stricter than the
MIT licence on the code and is the binding constraint on these artifacts.

The code in the GitHub repository is MIT, but the repository as a whole is not
usable commercially as it stands: the data preprocessing derives from
[NVIDIA EDM2](https://github.com/NVlabs/edm2) under CC BY-NC-SA 4.0, and
several vendored encoders and samplers trace back to Meta projects under
CC BY-NC 4.0. The repository's `NOTICE` lists this component by component.

These weights are also derivative of upstream models, each carrying its own
terms that continue to apply and that you should verify for your use:

| Component | Upstream |
|---|---|
| `f8d4` VAE initialization | [stabilityai/sd-vae-ft-ema](https://huggingface.co/stabilityai/sd-vae-ft-ema) |
| `f16d32` VAE initialization | VA-VAE, from [LightningDiT](https://github.com/hustvl/LightningDiT) |
| `f8d16` VAE initialization | FLUX.1 VAE, [black-forest-labs](https://huggingface.co/black-forest-labs) |
| Text encoder | [OpenAI CLIP](https://github.com/openai/CLIP) |
| Prior architecture lineage | [REPA](https://github.com/sihyun-yu/REPA), [REPA-E](https://github.com/End2End-Diffusion/REPA-E), and DiT/SiT before them |

Training datasets carry their own terms independently of this licence, and
those terms may further restrict what you may do with a model trained on them.

## Security

A `.pt` checkpoint is a pickle: loading one executes arbitrary Python. These
checkpoints store an `args` namespace alongside the tensors, so they cannot be
loaded with `weights_only=True` and require `weights_only=False` on torch 2.6
and newer. Load checkpoints only from sources you trust, and verify you are
downloading from the official repository.

To report a security vulnerability, follow `SECURITY.md` in the code
repository: contact the ByteDance security team rather than opening a public
issue.

## Citation

```bibtex
@article{zheng2026genfirst,
  title={GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling},
  author={Zheng, Guangting and Zhang, Yiyuan and Yang, Tao and Chen, Yunpeng and Zhu, Rui and Deng, Jiajun and Zhang, Yanyong},
  journal={arXiv preprint arXiv:2608.29335},
  year={2026}
}
```

## Contact

Questions through [GitHub Issues](https://github.com/ByteDance-Seed/GenFirst/issues) or
mail to [zgt@mail.ustc.edu.cn](mailto:zgt@mail.ustc.edu.cn). Security reports
go to the address in `SECURITY.md`, not to Issues.