README.md

metadata

license: apache-2.0
pipeline_tag: text-to-image
library_name: diffusers

XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation

This repository contains the official model of the paper XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation.

Abstract

Achieving fine-grained control over subject identity and semantic attributes (pose, style, lighting) in text-to-image generation, particularly for multiple subjects, often undermines the editability and coherence of Diffusion Transformers (DiTs). Many approaches introduce artifacts or suffer from attribute entanglement. To overcome these challenges, we propose a novel multi-subject controlled generation model XVerse. By transforming reference images into offsets for token-specific text-stream modulation, XVerse allows for precise and independent control for specific subject without disrupting image latents or features. Consequently, XVerse offers high-fidelity, editable multi-subject image synthesis with robust control over individual subject characteristics and semantic attributes. This advancement significantly improves personalized and complex scene generation capabilities.

📖 Introduction

XVerse introduces a novel approach to multi-subject image synthesis, offering precise and independent control over individual subjects without disrupting the overall image latents or features. We achieve this by transforming reference images into offsets for token-specific text-stream modulation.

This innovation enables high-fidelity, editable image generation where you can robustly control both individual subject characteristics (identity) and their semantic attributes. XVerse significantly enhances capabilities for personalized and complex scene generation.

⚡️ Quick Start

Requirements and Installation

First, install the necessary dependencies:

# Create a conda environment named XVerse with Python version 3.10.16
conda create -n XVerse python=3.10.16 -y
# Activate the XVerse environment
conda activate XVerse
# Use pip to install the dependencies specified in requirements.txt
pip install -r requirements.txt

Next, download the required checkpoints:

cd checkpoints
bash ./download_ckpts.sh
cd ..

Important: You'll also need to download the face recognition model model_ir_se50.pth from InsightFace_Pytorch and place it directly into the ./checkpoints/ folder.

Local Gradio Demo

To run the interactive Gradio demo locally, execute the following command:

bash run_demo.sh

Input Settings Explained

The Gradio demo provides several parameters to control your image generation process:

• Prompt: The textual description guiding the image generation.
• Generated Height/Width: Use the sliders to set the shape of the output image.
• Weight_id/ip: Adjust these weight parameters. Higher values generally lead to better subject consistency but might slightly impact the naturalness of the generated image.
• latent_lora_scale and vae_lora_scale: Control the LoRA scale. Similar to Weight_id/ip, larger LoRA values can improve subject consistency but may reduce image naturalness.
• vae_skip_iter_before and vae_skip_iter_after: Configure VAE skip iterations. Skipping more steps can result in better naturalness but might compromise subject consistency.

Input Images

The demo provides detailed control over your input images:

• Expand Panel: Click "Input Image X" to reveal the options for each image.
• Upload Image: Click "Image X" to upload your desired reference image.
• Image Description: Enter a description in the "Caption X" input box. You can also click "Auto Caption" to generate a description automatically.
• Detection & Segmentation: Click "Det & Seg" to perform detection and segmentation on the uploaded image.
• Crop Face: Use "Crop Face" to automatically crop the face from the image.
• ID Checkbox: Check or uncheck "ID or not" to determine whether to use ID-related weights for that specific input image.

⚠️ Important Usage Notes:

• Prompt Construction: The main text prompt MUST include the exact text you entered in the Image Description field for each active image. Generation will fail if this description is missing from the prompt.
  • Example: If you upload two images and set their descriptions as "a man with red hair" (for Image 1) and "a woman with blue eyes" (for Image 2), your main prompt might be: "A a man with red hair walking beside a woman with blue eyes in a park."
  • You can then write your main prompt simply as: "ENT1 walking beside ENT2 in a park." The code will automatically replace these placeholders with the full description text before generation.
• Active Images: Only images in expanded (un-collapsed) panels will be fed into the model. Collapsed image panels are ignored.

Inference with XVerseBench

First, please download XVerseBench according to the contents in the assets folder. Then, when running inference, please execute the following command:

bash ./eval/eval_scripts/run_eval.sh

The script will automatically evaluate the model on the XVerseBench dataset and save the results in the ./results folder.

License

The code in this project is licensed under Apache 2.0; the dataset is licensed under CC0, subject to the intellectual property owned by Bytedance. Meanwhile, the dataset is adapted from dreambench++, you should also comply with the license of dreambench++.

Citation

If XVerse is helpful, please help to ⭐ the repo.

If you find this project useful for your research, please consider citing our paper:

@article{chen2025xverse,
  title={XVerse: Consistent Multi-Subject Control of Identity and Semantic Attributes via DiT Modulation},
  author={Chen, Bowen and Zhao, Mengyi and Sun, Haomiao and Chen, Li and Wang, Xu and Du, Kang and Wu, Xinglong},
  journal={arXiv preprint arXiv:2506.21416},
  year={2025}
}