| --- |
| license: mit |
| pipeline_tag: text-to-image |
| library_name: pixeldense |
| base_model: zehongma/PixelGen |
| base_model_relation: finetune |
| datasets: |
| - BLIP3o/BLIP3o-60k |
| language: |
| - en |
| tags: |
| - text-to-image |
| - pixel-diffusion |
| - flow-matching |
| - representation-alignment |
| - neurips-2026 |
| --- |
| |
| # PixelDense-XXL T2I |
|
|
| **PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion** (NeurIPS 2026) |
|
|
| [📄 Paper](https://arxiv.org/abs/2610.00483) | [🌐 Project Page](https://bio.lehanyang.info/PixelDense/) | [💻 Code](https://github.com/Hansxsourse/PixelDense) |
|
|
|  |
|
|
| PixelDense uses frozen dense-prediction models as representation-alignment targets for pixel-space diffusion. DINOv2 and SAM2 supervise a semantic projection stream, Depth Anything v2 and Metric3D v2 supervise a geometric projection stream, and a weight-space orthogonality penalty keeps the two streams in disjoint subspaces of the denoiser feature. The teachers and alignment projections are used only during training, so this checkpoint is a plain 1.1B pixel-space text-to-image denoiser with the same inference cost as PixelGen. |
|
|
| ## Model details |
|
|
| | | | |
| |---|---| |
| | Architecture | JiT-T2I pixel-space denoiser (PixelGen-XXL) | |
| | Parameters | 1.1B | |
| | Resolution | 512 × 512 | |
| | Text encoder | [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B) (not included; downloaded automatically) | |
| | Sampler | Adam LM solver, 25 steps, classifier-free guidance 4.0 | |
| | Base model | [PixelGen-XXL T2I](https://huggingface.co/zehongma/PixelGen) | |
| | Fine-tuning data | [BLIP3o-60k](https://huggingface.co/datasets/BLIP3o/BLIP3o-60k) | |
|
|
| See the paper for the full training recipe. |
|
|
| ## Files |
|
|
| | File | Size | Contents | |
| |---|---|---| |
| | `PixelDense_XXL_T2I.ckpt` | 4.5 GB | EMA denoiser weights, fp32 | |
|
|
| The checkpoint holds weights only: `{"state_dict": {"ema_denoiser.<param>": tensor}}`. It has no optimizer state or training metadata and loads with `torch.load(path, weights_only=True)`. Use it with the [PixelDense code](https://github.com/Hansxsourse/PixelDense): the denoiser has one learned embedding (`visual_type_embedding`) that upstream PixelGen does not have. |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/Hansxsourse/PixelDense |
| cd PixelDense |
| pip install -r requirements.txt |
| |
| hf download xsourse/PixelDense PixelDense_XXL_T2I.ckpt --local-dir ./ckpts |
| python main.py predict -c ./configs_t2i/pixeldense_inference_res512.yaml |
| ``` |
|
|
| Images are written to `./workdirs/exp_pixeldense_inference/images/predict/`. To use your own prompts, pass a text file with one prompt per line: |
|
|
| ```bash |
| python main.py predict -c ./configs_t2i/pixeldense_inference_res512.yaml \ |
| --data.pred_dataset.init_args.conditions ./prompts.txt |
| ``` |
|
|
| Inference needs only the denoiser and the Qwen3 text encoder; the dense teachers are not loaded. |
|
|
| ## Results |
|
|
| 512 × 512, from Table 1 of the paper. Higher is better for every metric. |
|
|
| | Model | GenEval Overall | DPG-Bench | HPS v2.1 | |
| |---|---|---|---| |
| | PixelGen (released) | 0.79 | – | 0.281 | |
| | PixelGen + fine-tune (DINOv2 REPA only) | 0.7927 | 78.7 | 0.280 | |
| | **PixelDense** | **0.8093** | **78.9** | **0.282** | |
|
|
| "PixelGen + fine-tune" is the matched baseline: the same fine-tuning run with only the original DINOv2 alignment. |
|
|
| ## Limitations |
|
|
| This is a research checkpoint. It generates at 512 × 512 only, and its outputs can reflect biases in the base model's training data and in BLIP3o-60k. |
|
|
| ## License |
|
|
| The PixelDense weights are released under the MIT License. They are fine-tuned from [PixelGen-XXL T2I](https://huggingface.co/zehongma/PixelGen), which is released under Apache-2.0. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @inproceedings{yang2026pixeldense, |
| title={PixelDense: Dense Prediction as Representation Alignment for Pixel Diffusion}, |
| author={Yang, Lehan and Qi, Daiqing and Zhang, Wenhao and Li, Avery and Yang, Yiqing and Li, Yifan and Kong, Yu and Zheng, Haitian and Zhang, Zhifei and Lin, Zhe and Jampani, Varun and Li, Sheng}, |
| booktitle={Advances in Neural Information Processing Systems}, |
| year={2026} |
| } |
| ``` |
|
|
| ## Acknowledgement |
|
|
| PixelDense is built on [PixelGen](https://github.com/Zehong-Ma/PixelGen) and [DeCo](https://github.com/Zehong-Ma/DeCo). The teachers come from [DINOv2](https://github.com/facebookresearch/dinov2), [SAM 2](https://github.com/facebookresearch/sam2), [Depth Anything V2](https://github.com/DepthAnything/Depth-Anything-V2) and [Metric3D](https://github.com/YvanYin/Metric3D). |
|
|
