Proteus-ID

Proteus-ID: ID-Consistent and Motion-Coherent Video Customization

   

Authors: Guiyu Zhang¹, Chen Shi¹, Zijian Jiang¹, Xunzhi Xiang², Jingjing Qian¹, Shaoshuai Shi³, Li Jiang†¹

¹ The Chinese University of Hong Kong, Shenzhen ² Nanjing University  ³ Voyager Research, Didi Chuxing

TODO

• Release arXiv technique report
• Release full codes
• Release dataset (coming soon)

🛠️ Requirements and Installation

Environment

# 0. Clone the repo
git clone --depth=1 https://github.com/grenoble-zhang/Proteus-ID.git

cd /nfs/dataset-ofs-voyager-research/guiyuzhang/Opensource/code/Proteus-ID-main

# 1. Create conda environment
conda create -n proteusid python=3.11.0
conda activate proteusid

# 3. Install PyTorch and other dependencies
# CUDA 12.6
pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu126
# 4. Install pip dependencies
pip install -r requirements.txt

Download Model

cd util
python download_weights.py
python down_raft.py

Once ready, the weights will be organized in this format:

🔦 ckpts/
├── 📂 face_encoder/
├── 📂 scheduler/
├── 📂 text_encoder/
├── 📂 tokenizer/
├── 📂 transformer/
├── 📂 vae/
├── 📄 configuration.json
├── 📄 model_index.json

🏋️ Training

# For single rank
bash train_single_rank.sh
# For multi rank
bash train_multi_rank.sh

🏄️ Inference

python inference.py --img_file_path assets/example_images/1.png --json_file_path assets/example_images/1.json

BibTeX

If you find our work useful in your research, please consider citing our paper:

@article{zhang2025proteus,
  title={Proteus-ID: ID-Consistent and Motion-Coherent Video Customization},
  author={Zhang, Guiyu and Shi, Chen and Jiang, Zijian and Xiang, Xunzhi and Qian, Jingjing and Shi, Shaoshuai and Jiang, Li},
  journal={arXiv preprint arXiv:2506.23729},
  year={2025}
}

Acknowledgement

Thansk for these excellent opensource works and models: CogVideoX; ConsisID; diffusers.