Add files using upload-large-folder tool

MANIFEST.md

@@ -12,6 +12,12 @@ checkpoints used for the paper results, not the later validation runs.
   ImageNet-512 results.
 - `vae/e2e-flux-vae-sit-imagenet-s2/`: T2I variant A VAE (FLUX.1 init).
 - `vae/e2e-flux-vae-mmdit-l2-t2i-s2/`: T2I variant B VAE (FLUX.1 init).
+- `vae/qwenvit-siglip-b256-s2/`: Qwen3-VL ViT + ViT decoder VAE shaped with
+  E2E + SigLIP supervision (Table 9).
+- `vae/qwenvit-nlltrunk-s2/`: Qwen3-VL ViT + ViT decoder VAE shaped with
+  E2E + VLM-NLL supervision (Table 9).
+- `vae/e2e-joint-gan-l2-330k/`: Jointly E2E-shaped FLUX.1 image VAE + Cola text
+  VAE (Table 10).
 - `vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/`: 32-channel f8d32 E2E FLUX.2 VAE
   (three-stage aspect-ratio-bucketed E2E + decoder finetune).
 
@@ -21,16 +27,19 @@ Every VAE directory contains:
 - `<name>-latents-stats.pt`: required latent normalization statistics.
 - `<name>-discriminator-ckpt.pt`: only needed to resume end-to-end training.
 
-In addition, `vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/` includes:
+In addition:
 
-- `e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors` and `diffusion_pytorch_model.safetensors`:
-  diffusers-compatible safetensors weights.
-- `config.json` and `vae_params.json`: diffusers configuration and explicit per-channel
-  `scaling_factor` and `shift_factor`:
-  - `--vae_pth=vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors`
-  - `--model.vae.scaling_factor=[0.887221,0.886926,0.887096,0.881914,0.886901,0.887307,0.886685,0.886833,0.886540,0.886646,0.887031,0.887073,0.887381,0.886768,0.887299,0.885674,0.886466,0.886795,0.886979,0.886519,0.886949,0.886949,0.887036,0.886828,0.887344,0.887004,0.886816,0.886799,0.887354,0.887523,0.887012,0.886643]`
-  - `--model.vae.shift_factor=[-0.000623,0.000143,0.000282,0.001599,-0.001093,-0.000477,0.000855,-0.000830,-0.000883,0.001076,-0.000417,0.000200,0.000626,-0.000649,-0.000137,0.001453,-0.000816,-0.000307,-0.000290,0.000662,0.000484,0.000993,0.000574,-0.000139,-0.000914,0.000578,-0.000691,-0.000323,0.000619,-0.000606,-0.000372,0.000437]`
-- `e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k-stage2-bn-latents-stats.pt`: Stage-2 70k SiT EMA BatchNorm statistics.
+- `vae/e2e-joint-gan-l2-330k/` includes `e2e-joint-gan-l2-330k-text-vae.pt`
+  (standalone E2E-reshaped Cola text VAE state dict).
+- `vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/` includes:
+  - `e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors` and `diffusion_pytorch_model.safetensors`:
+    diffusers-compatible safetensors weights.
+  - `config.json` and `vae_params.json`: diffusers configuration and explicit per-channel
+    `scaling_factor` and `shift_factor`:
+    - `--vae_pth=vae/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k/e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k.safetensors`
+    - `--model.vae.scaling_factor=[0.887221,0.886926,0.887096,0.881914,0.886901,0.887307,0.886685,0.886833,0.886540,0.886646,0.887031,0.887073,0.887381,0.886768,0.887299,0.885674,0.886466,0.886795,0.886979,0.886519,0.886949,0.886949,0.887036,0.886828,0.887344,0.887004,0.886816,0.886799,0.887354,0.887523,0.887012,0.886643]`
+    - `--model.vae.shift_factor=[-0.000623,0.000143,0.000282,0.001599,-0.001093,-0.000477,0.000855,-0.000830,-0.000883,0.001076,-0.000417,0.000200,0.000626,-0.000649,-0.000137,0.001453,-0.000816,-0.000307,-0.000290,0.000662,0.000484,0.000993,0.000574,-0.000139,-0.000914,0.000578,-0.000691,-0.000323,0.000619,-0.000606,-0.000372,0.000437]`
+  - `e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k-stage2-bn-latents-stats.pt`: Stage-2 70k SiT EMA BatchNorm statistics.
 
 Keep each VAE and its latent statistics together.
 
@@ -65,16 +74,36 @@ GenEval/DPG generation scripts. Pass the matching VAE, latent statistics and
 `--text-encoder-name openai/clip-vit-large-patch14`; no checkpoint conversion
 is required.
 
+## Shared visual latents (generation + understanding, Table 9)
+
+- `shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt`, paired with
+  `vae/qwenvit-siglip-b256-s2/` (`0.90` GenEval, `83.76` DPG-Bench, `79.25%` IN1K linear probe).
+- `shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt`, paired with
+  `vae/qwenvit-nlltrunk-s2/` (`0.89` GenEval, `83.49` DPG-Bench, `81.69%` IN1K linear probe).
+
+Directly accepted by `Prior_only/infer_t2i.py` and `Prior_only/gen_for_bench.py`
+with `--vae-arch qwenvit --vae-vit-path Qwen/Qwen3-VL-2B-Instruct`.
+
+## Unified text-image modeling (`JointMMDiT-XL/2`, Table 10)
+
+- `unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt`: Phase-3
+  512x512 SFT checkpoint for text-to-image (`86.57` GenEval, `72.68` DPG-Bench).
+- `unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512.pt`: Phase-2
+  512x512 checkpoint before SFT for image-to-text COCO captioning (`33.28` CIDEr,
+  `57.97` CLIPScore).
+- Both pair with `vae/e2e-joint-gan-l2-330k/` (`--image-vae-ckpt vae/e2e-joint-gan-l2-330k/e2e-joint-gan-l2-330k.pt`;
+  the reshaped text VAE weights are already embedded inside `ckpt["text_vae"]`).
+
 ## Checkpoint formats produced by the repository
 
-- `train_prior_only.py` and `train_t2i_frozen.py` save final checkpoints in
-  direct evaluation format.
+- `train_prior_only.py`, `train_t2i_frozen.py`, and `train_joint_frozen.py` save
+  final checkpoints in direct evaluation format.
 - Class-conditional `train_e2e.py` checkpoints can be used directly by the
   top-level `generate.py`; all `train_e2e.py`, `train_e2e_vae.py`, and
   `train_decoder_ft.py` checkpoints can be evaluated directly for reconstruction.
   Before using either checkpoint's VAE with a separate prior, run
-  `Prior_only/scripts/run_prior_only_pipeline.sh extract`, which calls
-  `save_vae_weights.py` to produce a standalone VAE bundle.
+  `Prior_only/scripts/run_prior_only_pipeline.sh extract` (or `save_joint_vae_weights.py`
+  for `train_joint_e2e.py`), which produces a standalone VAE bundle.
 - A downloaded VAE bundle is already extracted and must not be extracted again.
 
 Weights are released under CC BY-NC 4.0. See the repository's

MODEL_CARD.md

@@ -84,6 +84,23 @@ rewriting. The two variants differ only in which VAE they were trained on.
 | EiT T2I, variant A | 512x512 | **0.8987** | **82.60** |
 | EiT T2I, variant B | 512x512 | 0.8920 | 82.06 |
 
+### Shared visual latents (generation + understanding)
+
+`MMDiT-XL/2` on a frozen Qwen3-VL ViT autoencoder (`--vae-arch qwenvit`) shaped end-to-end with a generative prior plus representation supervision (`SigLIP` or `VLM-NLL`, Table 9 in the paper).
+
+| Checkpoint | VAE bundle | Supervision | Resolution | rFID ↓ | PSNR ↑ | GenEval ↑ | DPG-Bench ↑ | Linear Probe Top-1 (%) ↑ |
+|---|---|---|---|---|---|---|---|---|
+| `shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt` | `vae/qwenvit-siglip-b256-s2/` | E2E + SigLIP | 512x512 | **0.39** | **27.64** | **0.90** | **83.76** | 79.25 |
+| `shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt` | `vae/qwenvit-nlltrunk-s2/` | E2E + VLM-NLL | 512x512 | 0.46 | 27.19 | 0.89 | 83.49 | **81.69** |
+
+### Unified text-image modeling (`JointMMDiT-XL/2`)
+
+Bidirectional block-causal `JointMMDiT-XL/2` trained on the jointly E2E-shaped text VAE (Cola) and image VAE (FLUX.1) (`vae/e2e-joint-gan-l2-330k/`, Table 10 in the paper). The phase-3 SFT checkpoint (`-512-sft.pt`) is used for text-to-image (GenEval / DPG-Bench), and the phase-2 pre-SFT checkpoint (`-512.pt`) is used for image-to-text COCO captioning.
+
+| Checkpoints | Setting | GenEval ↑ | DPG-Bench ↑ | CIDEr ↑ | BLEU-4 ↑ | METEOR ↑ | ROUGE-L ↑ | SPICE ↑ | CLIPScore ↑ |
+|---|---|---|---|---|---|---|---|---|---|
+| `unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt` (T2I) & `eit-joint-e2e-text-e2e-img-512.pt` (I2T) | E2E-joint text + image VAEs | **86.57** | **72.68** | **33.28** | **7.76** | **17.74** | **34.16** | **11.81** | **57.97** |
+
 ### End-to-end VAEs
 
 Reconstruction quality of the posterior mean, measured at the resolution the
@@ -95,6 +112,9 @@ VAE is used at.
 | `f16d32` | 16x, 32ch | VA-VAE (LightningDiT) | 0.36 | 25.82 | 0.74 | 0.12 |
 | `f8d16` variant A | 8x, 16ch | FLUX.1 VAE | 0.23 | 29.43 | 0.85 | 0.07 |
 | `f8d16` variant B | 8x, 16ch | FLUX.1 VAE | 0.21 | 29.37 | 0.85 | 0.07 |
+| `qwenvit-siglip-b256-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (SigLIP) | 0.39 | 27.64 | – | – |
+| `qwenvit-nlltrunk-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (VLM-NLL) | 0.46 | 27.19 | – | – |
+| `e2e-joint-gan-l2-330k` | 8x, 16ch (image) + 16d (text) | FLUX.1 VAE + Cola Text VAE | – | – | – | – |
 | `f8d32` (`e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, @512) | 8x, 32ch | FLUX.2 VAE | **0.041** | **36.46** | **0.953** | **0.026** |
 
 **A VAE and its latent statistics must be kept together.** The prior was

README.md

@@ -84,6 +84,23 @@ rewriting. The two variants differ only in which VAE they were trained on.
 | EiT T2I, variant A | 512x512 | **0.8987** | **82.60** |
 | EiT T2I, variant B | 512x512 | 0.8920 | 82.06 |
 
+### Shared visual latents (generation + understanding)
+
+`MMDiT-XL/2` on a frozen Qwen3-VL ViT autoencoder (`--vae-arch qwenvit`) shaped end-to-end with a generative prior plus representation supervision (`SigLIP` or `VLM-NLL`, Table 9 in the paper).
+
+| Checkpoint | VAE bundle | Supervision | Resolution | rFID ↓ | PSNR ↑ | GenEval ↑ | DPG-Bench ↑ | Linear Probe Top-1 (%) ↑ |
+|---|---|---|---|---|---|---|---|---|
+| `shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt` | `vae/qwenvit-siglip-b256-s2/` | E2E + SigLIP | 512x512 | **0.39** | **27.64** | **0.90** | **83.76** | 79.25 |
+| `shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt` | `vae/qwenvit-nlltrunk-s2/` | E2E + VLM-NLL | 512x512 | 0.46 | 27.19 | 0.89 | 83.49 | **81.69** |
+
+### Unified text-image modeling (`JointMMDiT-XL/2`)
+
+Bidirectional block-causal `JointMMDiT-XL/2` trained on the jointly E2E-shaped text VAE (Cola) and image VAE (FLUX.1) (`vae/e2e-joint-gan-l2-330k/`, Table 10 in the paper). The phase-3 SFT checkpoint (`-512-sft.pt`) is used for text-to-image (GenEval / DPG-Bench), and the phase-2 pre-SFT checkpoint (`-512.pt`) is used for image-to-text COCO captioning.
+
+| Checkpoints | Setting | GenEval ↑ | DPG-Bench ↑ | CIDEr ↑ | BLEU-4 ↑ | METEOR ↑ | ROUGE-L ↑ | SPICE ↑ | CLIPScore ↑ |
+|---|---|---|---|---|---|---|---|---|---|
+| `unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt` (T2I) & `eit-joint-e2e-text-e2e-img-512.pt` (I2T) | E2E-joint text + image VAEs | **86.57** | **72.68** | **33.28** | **7.76** | **17.74** | **34.16** | **11.81** | **57.97** |
+
 ### End-to-end VAEs
 
 Reconstruction quality of the posterior mean, measured at the resolution the
@@ -95,6 +112,9 @@ VAE is used at.
 | `f16d32` | 16x, 32ch | VA-VAE (LightningDiT) | 0.36 | 25.82 | 0.74 | 0.12 |
 | `f8d16` variant A | 8x, 16ch | FLUX.1 VAE | 0.23 | 29.43 | 0.85 | 0.07 |
 | `f8d16` variant B | 8x, 16ch | FLUX.1 VAE | 0.21 | 29.37 | 0.85 | 0.07 |
+| `qwenvit-siglip-b256-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (SigLIP) | 0.39 | 27.64 | – | – |
+| `qwenvit-nlltrunk-s2` | 8x, 16ch | Qwen3-VL ViT + ViT decoder (VLM-NLL) | 0.46 | 27.19 | – | – |
+| `e2e-joint-gan-l2-330k` | 8x, 16ch (image) + 16d (text) | FLUX.1 VAE + Cola Text VAE | – | – | – | – |
 | `f8d32` (`e2e-vae-flux2-sitxl2-auxkl2p5e8-s3-20k`, @512) | 8x, 32ch | FLUX.2 VAE | **0.041** | **36.46** | **0.953** | **0.026** |
 
 **A VAE and its latent statistics must be kept together.** The prior was

shared-latents/siglip/eit-qwenvit-siglip-t2i-512.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:201df4911e369f349ae8be1558b62d731b9d7eefe653577af15cffc66db69231
+size 21368379236

shared-latents/vlm-nll/eit-qwenvit-vlm-nll-t2i-512.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:25ebab2f9c7215951d5cb1da206c547a967974b3b606bf3d14c2f013bc67ed70
+size 21368379236

unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512-sft.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:df8fe9ee136f219cfb7127cb79d75afabf99a75fb4590545fe7076801f884c90
+size 23683508731

unified-text-image/e2e-joint/eit-joint-e2e-text-e2e-img-512.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79c7e568779b5452b0ca8ba607dced66cd12c40d920a6f4d89dd4eb29b634bd4
+size 23683508923

vae/e2e-joint-gan-l2-330k/e2e-joint-gan-l2-330k-discriminator-ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:56c34514c52a0725d1cc5179c36acbadb2db02ce749a7ecf9b6c8a84b0e95dfa
+size 11077422

vae/e2e-joint-gan-l2-330k/e2e-joint-gan-l2-330k-latents-stats.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1ec7b89bcf35c012a469f79580574526f9239ebe5a948ca51c49494dda35fa31
+size 1856

vae/e2e-joint-gan-l2-330k/e2e-joint-gan-l2-330k-text-vae.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6b90cd677f4986ec134b5798c4c5796d093fc95e2012a2e70dca056ee496ce10
+size 2007536503

vae/e2e-joint-gan-l2-330k/e2e-joint-gan-l2-330k.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8bdc222ecbd1e223b024ff37f2badb5538e6cf26bc2f1015a2e1aa2c744116a7
+size 335376026

vae/qwenvit-nlltrunk-s2/qwenvit-nlltrunk-s2-discriminator-ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:11eeffc2cde36582ed44b95d9d7810c71a2b02b63afc506038369b19d0337ed9
+size 11077690

vae/qwenvit-nlltrunk-s2/qwenvit-nlltrunk-s2-latents-stats.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6eb9827ffd752e757a419c4d32640fe01b38c8077421b4b014399a7e4d5d0755
+size 1844

vae/qwenvit-nlltrunk-s2/qwenvit-nlltrunk-s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:728487ea9ce81089849a6a60bf297fd0838de9dab80352388ed412e754440a0b
+size 2237629562

vae/qwenvit-siglip-b256-s2/qwenvit-siglip-b256-s2-discriminator-ckpt.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7cbf385cf41ab71f51ddca55935efd54710c98ba4f0bb56f0be95c42a6cba09f
+size 11077768

vae/qwenvit-siglip-b256-s2/qwenvit-siglip-b256-s2-latents-stats.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33df17bd8c0e54834cc3c69f9bf470d77bc3eb9bd4d27fa01afee9952af22479
+size 1862

vae/qwenvit-siglip-b256-s2/qwenvit-siglip-b256-s2.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1a7b821df9bbdc7d1006c27282dfdeb542831d3fa6d00416e74ae78aa0172ef
+size 2237631066