--- license: cc-by-nc-4.0 library_name: pytorch pipeline_tag: text-to-image tags: - diffusion - latent-diffusion - end-to-end-training - variational-autoencoder - text-to-image - class-conditional-image-generation - sit - mmdit --- # GenFirst Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling. - Paper: [arXiv:2608.29335](https://arxiv.org/abs/2608.29335) - Code: [github.com/ByteDance-Seed/GenFirst](https://github.com/ByteDance-Seed/GenFirst) - Project page: [tom-zgt.github.io/GenFirst](https://tom-zgt.github.io/GenFirst/) These weights are released under **CC BY-NC 4.0: non-commercial research use only**, which is stricter than the MIT licence on the code. See [Licence and third-party terms](#licence-and-third-party-terms) before use. ## What this is GenFirst trains a latent VAE jointly with a generative prior without the latent collapse that usually makes such end-to-end training unstable. The method is a two-phase schedule over the prior and entropy weights: phase 1 lets the prior shape the latent geometry, phase 2 drops both weights tenfold so the decoder recovers detail. The result is a latent space that a diffusion prior fits well while remaining a good autoencoder. This release covers the **EiT** (diffusion-prior) instantiation. The autoregressive **EAR** instantiation from the paper is not released. Two kinds of artifact are published, and they are used together: 1. **End-to-end VAEs** — the encoder/decoder produced by the end-to-end stage, each shipped with its latent normalization statistics. 2. **Generative priors** — SiT models for class-conditional generation, and MMDiT models for text-to-image, each trained on a frozen VAE from (1). ## Model details | | | |---|---| | Developed by | Guangting Zheng, Yiyuan Zhang, Tao Yang, Yunpeng Chen, Rui Zhu, Jiajun Deng, Yanyong Zhang | | Model type | Latent diffusion (flow matching / stochastic interpolant) with a jointly trained VAE | | Prior architecture | SiT-XL and MMDiT-XL: 28 blocks, width 1152, 16 attention heads (~675M parameters) | | Text encoder (text-to-image) | CLIP-L, frozen, not modified by us | | Precision | Weights released in fp32; sampling tested in bf16 | | Framework | PyTorch | | Inference code | Required — see [Usage](#usage). These are raw `.pt` checkpoints, not `diffusers` pipelines. | ### Class-conditional priors `SiT-XL` on a frozen end-to-end VAE. Conditioning is a single integer class label in `[0, 999]` following the standard 1000-class label ordering; there is no text input. Epoch counts are prior-only training epochs. | Checkpoint | Resolution | VAE | gFID | IS | gFID (guided) | IS (guided) | Download | |---|---|---|---|---|---|---|---| | EiT, 80 epochs | 256x256 | `f16d32` | 2.79 | 175.4 | 1.38 | 271.3 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-80ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) | | EiT, 480 epochs | 256x256 | `f16d32` | 1.60 | 216.8 | 0.988 | 270.7 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-480ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) | | EiT, 800 epochs | 256x256 | `f16d32` | **1.45** | 226.7 | **0.969** | 283.4 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-800ep.pt) · [weak-20ep](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet256/eit-imagenet256-weak-20ep.pt) | | EiT, 80 epochs | 512x512 | `f16d32` | – | – | 1.63 | 278.9 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet512/eit-imagenet512-80ep.pt) | | EiT, 400 epochs | 512x512 | `f16d32` | – | – | **1.23** | 286.1 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/class-conditional/imagenet512/eit-imagenet512-400ep.pt) | The 512 models reuse the 256 `f16d32` VAE unchanged and frozen; only the prior is trained at 512. ### Text-to-image priors `MMDiT-XL/2` with a frozen CLIP-L text encoder, on a frozen end-to-end FLUX VAE, at 512x512. Scores are overall GenEval and DPG-Bench, with no prompt rewriting. The two variants differ only in which VAE they were trained on. | Checkpoint | Resolution | GenEval | DPG-Bench | Download | |---|---|---|---|---| | EiT T2I, variant A | 512x512 | **0.8987** | **82.60** | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/text-to-image/variant-a/eit-t2i-variant-a-512.pt) · [VAE](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-sit-imagenet-s2) | | EiT T2I, variant B | 512x512 | 0.8920 | 82.06 | [checkpoint](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/text-to-image/variant-b/eit-t2i-variant-b-512.pt) · [VAE](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-mmdit-l2-t2i-s2) | ### Shared visual latents (generation + understanding) `MMDiT-XL/2` on a frozen Qwen3-VL ViT autoencoder (`--vae-arch qwenvit`) shaped end-to-end with a generative prior plus representation supervision (`SigLIP` or `VLM-NLL`, Table 9 in the paper). | Checkpoint | VAE bundle | Supervision | Resolution | rFID ↓ | PSNR ↑ | GenEval ↑ | DPG-Bench ↑ | Linear Probe Top-1 (%) ↑ | |---|---|---|---|---|---|---|---|---| | [`shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt) | [`vae/qwenvit-siglip-b256-s2/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-siglip-b256-s2) | E2E + SigLIP | 512x512 | **0.39** | **27.64** | **0.90** | **83.76** | 79.25 | | [`shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt) | [`vae/qwenvit-nlltrunk-s2/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-nlltrunk-s2) | E2E + VLM-NLL | 512x512 | 0.46 | 27.19 | 0.89 | 83.49 | **81.69** | ### Unified text-image modeling (`JointMMDiT-XL/2`) Bidirectional block-causal `JointMMDiT-XL/2` trained on the jointly E2E-shaped text VAE (Cola) and image VAE (FLUX.1) (`vae/e2e-joint-gan-l2-330k/`, Table 10 in the paper). The phase-3 SFT checkpoint (`-512-sft.pt`) is used for text-to-image (GenEval / DPG-Bench), and the phase-2 pre-SFT checkpoint (`-512.pt`) is used for image-to-text COCO captioning. | Checkpoints | Setting | GenEval ↑ | DPG-Bench ↑ | CIDEr ↑ | BLEU-4 ↑ | METEOR ↑ | ROUGE-L ↑ | SPICE ↑ | CLIPScore ↑ | |---|---|---|---|---|---|---|---|---|---| | [`eit-joint-e2e-text-e2e-img-512-sft.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt) (T2I) & [`eit-joint-e2e-text-e2e-img-512.pt`](https://huggingface.co/ByteDance-Seed/GenFirst/blob/main/unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512.pt) (I2T) | E2E-joint text + image VAEs ([`vae/e2e-joint-gan-l2-330k/`](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-joint-gan-l2-330k)) | **86.57** | **72.68** | **33.28** | **7.76** | **17.74** | **34.16** | **11.81** | **57.97** | ### End-to-end VAEs Reconstruction quality of the posterior mean, measured at the resolution the VAE is used at. | VAE | Downsample / channels | Initialized from | rFID | PSNR | SSIM | LPIPS | Download | |---|---|---|---|---|---|---|---| | `f8d4` | 8x, 4ch | SD-VAE (`stabilityai/sd-vae-ft-ema`) | **0.57** | 25.13 | 0.71 | 0.14 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f8d4) | | `f16d32` (decoder-ft, @256) | 16x, 32ch | VA-VAE (LightningDiT) | 0.36 | 25.82 | 0.74 | 0.12 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f16d32) | | `f16d32` (no dec-ft, @512) | 16x, 32ch | VA-VAE (LightningDiT) | – | – | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-f16d32-nodecft) | | `f8d16` variant A | 8x, 16ch | FLUX.1 VAE | 0.23 | 29.43 | 0.85 | 0.07 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-sit-imagenet-s2) | | `f8d16` variant B | 8x, 16ch | FLUX.1 VAE | 0.21 | 29.37 | 0.85 | 0.07 | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-flux-vae-mmdit-l2-t2i-s2) | | `qwenvit-siglip-b256-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (SigLIP) | 0.39 | 27.64 | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-siglip-b256-s2) | | `qwenvit-nlltrunk-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (VLM-NLL) | 0.46 | 27.19 | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/qwenvit-nlltrunk-s2) | | `e2e-joint-gan-l2-330k` | 8x, 16ch (image) + 16d (text) | FLUX.1 VAE + Cola Text VAE | – | – | – | – | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-joint-gan-l2-330k) | | `f8d32` (`e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, @512) | 8x, 32ch | FLUX.2 VAE | **0.041** | **36.46** | **0.953** | **0.026** | [Hugging Face](https://huggingface.co/ByteDance-Seed/GenFirst/tree/main/vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k) | **A VAE and its latent statistics must be kept together.** The prior was trained on latents normalized by those exact statistics; pairing a VAE with the wrong statistics file produces noise, not a degraded image. Each VAE directory is laid out as: ``` / .pt VAE weights .safetensors VAE weights in diffusers safetensors format (for FLUX.2 release) -latents-stats.pt latent normalization (scale + bias) -discriminator-ckpt.pt discriminator, only needed to resume end-to-end training config.json / vae_params.json diffusers config and explicit scaling_factor / shift_factor (for FLUX.2 release) ``` For `vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, the per-channel `scaling_factor` and `shift_factor` parameters are: ```bash --vae_pth=vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors --model.vae.scaling_factor=[0.887221,0.886926,0.887096,0.881914,0.886901,0.887307,0.886685,0.886833,0.886540,0.886646,0.887031,0.887073,0.887381,0.886768,0.887299,0.885674,0.886466,0.886795,0.886979,0.886519,0.886949,0.886949,0.887036,0.886828,0.887344,0.887004,0.886816,0.886799,0.887354,0.887523,0.887012,0.886643] --model.vae.shift_factor=[-0.000623,0.000143,0.000282,0.001599,-0.001093,-0.000477,0.000855,-0.000830,-0.000883,0.001076,-0.000417,0.000200,0.000626,-0.000649,-0.000137,0.001453,-0.000816,-0.000307,-0.000290,0.000662,0.000484,0.000993,0.000574,-0.000139,-0.000914,0.000578,-0.000691,-0.000323,0.000619,-0.000606,-0.000372,0.000437] ``` ## Usage These checkpoints need the training repository; there is no `diffusers` integration and no `from_pretrained` entry point. ```bash git clone https://github.com/ByteDance-Seed/GenFirst cd GenFirst pip install -r requirements.txt # install torch first ``` Class-conditional sampling, writing a small batch of images rather than a full evaluation run: ```bash cd Prior_only python generate.py \ --model SiT-XL/1 --resolution 256 \ --ckpt /path/to/prior.pt \ --vae-arch f16d32 \ --vae-ckpt /path/to/e2e-f16d32.pt \ --latents-stats /path/to/e2e-f16d32-latents-stats.pt \ --num-fid-samples 64 --label-sampling random \ --cfg-scale 1.5 --sample-dir samples ``` `--model` must match the VAE: `SiT-XL/2` for `f8d4`, `SiT-XL/1` for `f16d32`. Text-to-image sampling, the guidance settings that reproduce the numbers above, and the full evaluation harness are documented in the repository under `docs/`. ## Evaluation Reported metrics and their protocols: - **gFID / IS** — 50,000 generated samples scored with [torch-fidelity](https://github.com/LTH14/torch-fidelity) against the reference statistics used by [MAR](https://github.com/LTH14/mar), at the stated resolution. Sampling is SDE with 250 steps. Guided rows use [autoguidance](https://arxiv.org/abs/2406.02507) (pointing away from an early checkpoint of the same run) with CFG on top, except the 512 rows, which use plain CFG. - **rFID / PSNR / SSIM / LPIPS** — posterior-mean reconstructions over a 50,000-image validation split, following [latent-diffusion](https://github.com/CompVis/latent-diffusion). rFID here is measured against those validation images, not a training-set reference, so it is not comparable to numbers computed the other way. - **GenEval / DPG-Bench** — overall scores, Euler sampling with 50 steps and CFG 7.0 on the interval `[0.35, 1.0]`, no prompt rewriting. Numbers were produced with the versions pinned in `environment.yml`. FID is sensitive to reference statistics and to the resizing path, so comparisons against other work are only meaningful under the same protocol. ## Training Training proceeds in two stages. The end-to-end stage trains the VAE jointly with a prior under the two-phase GenFirst schedule (global batch 256; 400k steps at the high weight, then 200k at the low one). The prior-only stage then freezes the VAE, caches its latents, and trains from scratch the prior that is actually evaluated (4M steps at 256, 2M at 512). An optional 10k-step decoder-only finetune between the stages improves unguided gFID by roughly 0.2. Text-to-image follows the same structure, with an `MMDiT-L/2` prior shaping the latent during the end-to-end stage and the released `MMDiT-XL/2` trained on the frozen result. Training data consists of publicly available datasets. Individual corpora are not enumerated here; each is subject to its own licence and terms of use, which govern independently of the licence on these weights. ## Intended use Intended for research: reproducing the paper, studying end-to-end VAE and prior training, latent-space analysis, and use as a baseline or initialization for further non-commercial research. **Out of scope.** Any commercial use. Generating content about real, identifiable people. Generating sexual, violent, hateful, or otherwise harmful imagery. Any use presenting outputs as real photographs or as factual records. Any safety-critical or decision-making pipeline. These models produce synthetic images with no factual grounding and must not be treated as a source of truth. No safety filter, watermarking, or content classifier is included. Anyone deploying these weights in a setting where outputs reach other people is responsible for adding those controls. ## Limitations and biases - The class-conditional models generate from a fixed 1000-class label set and cannot follow text. Their outputs reflect the visual distribution and the category taxonomy they were fit to, including the geographic and cultural skew documented for that style of curated image collection. - The text-to-image models inherit the biases of both their training data and the frozen CLIP-L text encoder. Expect uneven representation across demographics, professions, and cultures, and expect prompt-following to degrade on long, compositional, or counting-heavy prompts. GenEval and DPG-Bench scores measure prompt alignment on those benchmarks only and do not certify safety or fairness. - Text rendering, hands, and fine structure remain unreliable, as with comparable models at this scale. - Metrics reported here are distribution-level. A good FID says nothing about the quality or appropriateness of any single sample. - The 512 class-conditional models are reported with guidance only; their unguided behaviour is untuned. - A VAE paired with mismatched latent statistics fails silently in the sense that it produces output rather than an error. Check the pairing. ## Licence and third-party terms **Weights: [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — non-commercial research use only, with attribution.** This is stricter than the MIT licence on the code and is the binding constraint on these artifacts. The code in the GitHub repository is MIT, but the repository as a whole is not usable commercially as it stands: the data preprocessing derives from [NVIDIA EDM2](https://github.com/NVlabs/edm2) under CC BY-NC-SA 4.0, and several vendored encoders and samplers trace back to Meta projects under CC BY-NC 4.0. The repository's `NOTICE` lists this component by component. These weights are also derivative of upstream models, each carrying its own terms that continue to apply and that you should verify for your use: | Component | Upstream | |---|---| | `f8d4` VAE initialization | [stabilityai/sd-vae-ft-ema](https://huggingface.co/stabilityai/sd-vae-ft-ema) | | `f16d32` VAE initialization | VA-VAE, from [LightningDiT](https://github.com/hustvl/LightningDiT) | | `f8d16` VAE initialization | FLUX.1 VAE, [black-forest-labs](https://huggingface.co/black-forest-labs) | | Text encoder | [OpenAI CLIP](https://github.com/openai/CLIP) | | Prior architecture lineage | [REPA](https://github.com/sihyun-yu/REPA), [REPA-E](https://github.com/End2End-Diffusion/REPA-E), and DiT/SiT before them | Training datasets carry their own terms independently of this licence, and those terms may further restrict what you may do with a model trained on them. ## Security A `.pt` checkpoint is a pickle: loading one executes arbitrary Python. These checkpoints store an `args` namespace alongside the tensors, so they cannot be loaded with `weights_only=True` and require `weights_only=False` on torch 2.6 and newer. Load checkpoints only from sources you trust, and verify you are downloading from the official repository. To report a security vulnerability, follow `SECURITY.md` in the code repository: contact the ByteDance security team rather than opening a public issue. ## Citation ```bibtex @article{zheng2026genfirst, title={GenFirst: Generation Before Reconstruction for Stable End-to-End Latent Generative Modeling}, author={Zheng, Guangting and Zhang, Yiyuan and Yang, Tao and Chen, Yunpeng and Zhu, Rui and Deng, Jiajun and Zhang, Yanyong}, journal={arXiv preprint arXiv:2608.29335}, year={2026} } ``` ## Contact Questions through [GitHub Issues](https://github.com/ByteDance-Seed/GenFirst/issues) or mail to [zgt@mail.ustc.edu.cn](mailto:zgt@mail.ustc.edu.cn). Security reports go to the address in `SECURITY.md`, not to Issues.