README.md

---
license: mit
language:
  - en
  - zh
library_name: pytorch
tags:
  - multimodal
  - image-generation
  - image-editing
  - visual-understanding
  - vision-encoder
  - image-reconstruction
arxiv: 2608.08676
---

<p align="center">
  <img src="https://raw.githubusercontent.com/yjb6/UniSpace/main/project-page/assets/unispace-mark-abbr.svg" width="180" alt="UniSpace logo">
</p>

<h1 align="center">UniSpace</h1>

<p align="center">
  <b>Unified Visual Representation and Scalable Multimodal Modeling</b>
</p>

<p align="center">
  <a href="https://huggingface.co/papers/2608.08676">HF Paper</a> ·
  <a href="https://arxiv.org/abs/2608.08676">arXiv</a> ·
  <a href="https://github.com/yjb6/UniSpace">Code</a> ·
  <a href="https://yjb6.github.io/UniSpace/">Project Page</a>
</p>

UniSpace is a unified visual representation framework spanning image
understanding, reconstruction, generation, and instruction-based editing. This
repository provides the selected inference checkpoints and normalization
statistics associated with the paper.

## Overview

UniSpace contains two related components:

1. **Patch-reparameterized vision encoders.** PR-SigLIP2, PR-DINOv2, and
   PR-Qwen-ViT retain the semantic representation of pretrained vision
   encoders while adding the image detail needed for faithful reconstruction
   and high-quality generation.
2. **The UniSpace multimodal model.** A Qwen-based patch-reparameterized visual
   tokenizer is coupled with a Qwen3-8B Mixture-of-Transformers model for visual
   understanding, text-to-image generation, and image editing.

The public code release is inference and evaluation only. UniSpace training
code, optimizer states, private data pipelines, and cluster-specific files are
not included.

## Checkpoints

| Artifact | Role | Used for |
|---|---|---|
| `encoders/pr-siglip2-tokenizer.pt` | Unified PR-SigLIP2 tokenizer | Reconstruction and ImageNet generation |
| `encoders/pr-dinov2-tokenizer.pt` | PR-DINOv2 tokenizer | Reconstruction and generation |
| `encoders/pr-qwen-vit-tokenizer.pt` | PR-Qwen-ViT tokenizer | Reconstruction and UniSpace initialization |
| `encoders/pr-siglip2-dit.pt` | Class-conditional DiT | PR-SigLIP2 ImageNet generation |
| `encoders/pr-dinov2-dit.pt` | Class-conditional DiT | PR-DINOv2 ImageNet generation |
| `stats/*.pt` | Latent normalization statistics | Matching encoder evaluation configs |
| `unispace-sft-0012000/model-*.safetensors` | Final selected UniSpace SFT checkpoint (17 standard shards) | Understanding, generation, and editing |

The selected UniSpace model is SFT step `0012000`, initialized from stage-3
step `0060000`. Earlier training checkpoints are not required for inference.
File digests are provided in [`SHA256SUMS`](./SHA256SUMS).

## Download

Download the complete release snapshot:

```bash
hf download yjb6/UniSpace --local-dir checkpoints/UniSpace
```

Or download one artifact:

```bash
hf download yjb6/UniSpace \
  encoders/pr-dinov2-tokenizer.pt \
  --local-dir checkpoints/UniSpace
```

Expected layout:

```text
checkpoints/UniSpace/
├── encoders/
│   ├── pr-siglip2-tokenizer.pt
│   ├── pr-dinov2-tokenizer.pt
│   ├── pr-qwen-vit-tokenizer.pt
│   ├── pr-siglip2-dit.pt
│   └── pr-dinov2-dit.pt
├── stats/
│   ├── pr-siglip2-normalization-stats.pt
│   ├── pr-dinov2-normalization-stats.pt
│   └── pr-qwen-vit-normalization-stats.pt
└── unispace-sft-0012000/
    ├── model.safetensors.index.json
    └── model-00001-of-00017.safetensors ... model-00017-of-00017.safetensors
```

## Installation

```bash
git clone https://github.com/yjb6/UniSpace.git
cd UniSpace
conda env create -f environment.yml
conda activate rae
hf download yjb6/UniSpace --local-dir checkpoints/UniSpace
```

The verified environment uses Python 3.10, PyTorch 2.8.0, torchvision 0.23.0,
Transformers 4.57.3, and Accelerate 1.12.0. See the
[GitHub reproduction guide](https://github.com/yjb6/UniSpace#installation) for
dataset preparation, model dependencies, distributed launch commands, and
canonical evaluator setup.

## Reproduce encoder reconstruction

Prepare ImageNet-1K in its standard class-directory layout and set the paths
described in the code repository. For example:

```bash
cd patch-reparameterization

bash run_eval_only.sh configs/release/pr-siglip2-imagenet256.yaml \
  ../checkpoints/UniSpace/encoders/pr-siglip2-tokenizer.pt \
  --output-dir ../outputs/pr-siglip2-recon \
  --num-samples 50000 --batch-size 32 --no-zeroshot

bash run_eval_only.sh configs/release/pr-dinov2-imagenet256.yaml \
  ../checkpoints/UniSpace/encoders/pr-dinov2-tokenizer.pt \
  --output-dir ../outputs/pr-dinov2-recon \
  --num-samples 50000 --batch-size 64 --no-zeroshot

bash run_eval_only.sh configs/release/pr-qwen-vit-imagenet256.yaml \
  ../checkpoints/UniSpace/encoders/pr-qwen-vit-tokenizer.pt \
  --output-dir ../outputs/pr-qwen-vit-recon \
  --num-samples 50000 --batch-size 16 --no-zeroshot
```

Each run evaluates EMA weights and reports PSNR, SSIM, rFID, and sample count.
The complete generation and multimodal benchmark commands are documented in
the [main README](https://github.com/yjb6/UniSpace#reproduce-imagenet-generation).

## Results

### Patch-reparameterized encoders: reconstruction

ImageNet-1K validation, 256 × 256, 50,000 images:

| Encoder | PSNR ↑ | SSIM ↑ | rFID ↓ |
|---|---:|---:|---:|
| PR-SigLIP2 | 29.64 | 0.87 | 0.18 |
| PR-DINOv2 | 30.84 | 0.90 | 0.14 |
| PR-Qwen-ViT | 30.16 | 0.88 | 0.17 |

### Patch-reparameterized encoders: ImageNet generation

ImageNet-1K class-conditional generation, 256 × 256, 50,000 images:

| Encoder | CFG | gFID ↓ | sFID ↓ | IS ↑ | Precision ↑ | Recall ↑ |
|---|---:|---:|---:|---:|---:|---:|
| PR-SigLIP2 | 1.0 | 4.409 | 6.675 | 190.66 | 0.722 | 0.646 |
| PR-SigLIP2 | 1.2 | 2.799 | 6.055 | 248.05 | 0.777 | 0.611 |
| PR-DINOv2 | 1.0 | 2.100 | 5.377 | 216.97 | 0.779 | 0.637 |
| PR-DINOv2 | 1.2 | 1.877 | 4.888 | 274.16 | 0.822 | 0.605 |

### UniSpace generation

| Benchmark | Breakdown | Scores | Overall ↑ |
|---|---|---|---:|
| GenEval | Single / Two / Count / Color / Position / Attribute | 0.98 / 0.92 / 0.69 / 0.88 / 0.83 / 0.73 | 0.84 |
| DPG-Bench | Global / Entity / Attribute / Relation / Other | 84.80 / 92.26 / 90.00 / 94.97 / 88.80 | 86.49 |
| OneIG EN | Align / Text / Reason / Style / Diversity | 0.860 / 0.937 / 0.311 / 0.467 / 0.233 | 0.561 |
| OneIG ZH | Align / Text / Reason / Style / Diversity | 0.807 / 0.881 / 0.276 / 0.455 / 0.244 | 0.533 |

### UniSpace editing

| Benchmark | Breakdown | Scores | Overall ↑ |
|---|---|---|---:|
| ImgEdit | Add / Adjust / Extract / Replace / Remove / Background / Style / Hybrid / Action | 4.53 / 4.38 / 3.61 / 4.67 / 4.42 / 4.23 / 4.55 / 2.70 / 4.47 | 4.28 |
| GEdit EN | Semantic consistency / Perceptual quality | 8.287 / 7.055 | 7.407 |
| GEdit ZH | Semantic consistency / Perceptual quality | 8.270 / 6.998 | 7.382 |

Judge-based editing scores may vary with the evaluator endpoint and model
version. The clean ImgEdit rerun generated all 737 expected outputs and scored
4.25 overall; category differences from the paper result were within 0.30.

## Intended use and limitations

The release is intended for research on visual representation learning,
multimodal inference, image generation, image editing, and reproducible
benchmarking. Generated or edited images may contain factual, compositional,
text-rendering, or perceptual errors. Users should independently assess
outputs before using them in consequential settings. The release does not add
application-specific safety guarantees beyond those of its base models.

## Citation

```bibtex
@article{yan2026unispace,
  title   = {UniSpace: Unified Visual Representation and Scalable Multimodal Modeling},
  author  = {Yan, Jinbo and Qiao, Limeng and Qin, Jie and He, Jun-Yan and Wu, Feize and Wan, Guanglu},
  journal = {arXiv preprint arXiv:2608.08676},
  year    = {2026}
}
```

## License and third-party components

Repository code is distributed under the licenses included in the GitHub
release. Base-model and benchmark components retain their respective licenses
and terms. See
[`THIRD_PARTY.md`](https://github.com/yjb6/UniSpace/blob/main/THIRD_PARTY.md)
before redistribution or commercial use.