HumanAIGC-Team/UCM

UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models

Tian-Xing Xu^∗¹^, Zi-Xuan Wang^∗¹^, Guangyuan Wang^∗†²^, Li Hu^†²^, Zhongyi Zhang^³^, Peng Zhang^²^, Bang Zhang^‡²^, Song-Hai Zhang^‡¹^

¹Tsinghua University   ²Tongyi Lab, Alibaba   ³UCST

^∗Co-first authors   ^†Project leaders   ^‡Corresponding authors

   

📝 Introduction

We present UCM, a novel framework to explore 4D world of a reference image following the user-specified camera trajectory, which unifies long-term memory and precise camera control via a time-aware positional encoding warping mechanism.

Release Notes:

• [2026/08/07] 🔥🔥🔥UCM is released now, have fun!

🚀 Quick Start

Installation

1. Clone this repo:

git clone --recursive https://github.com/HumanAIGC/UCM.git

2. Install dependencies (please refer to requirements.txt):

pip install -r requirements.txt

Model Download

Model	Download Links
UCM	🤗 HuggingFace 🤖 ModelScope

Download models using huggingface-cli:

pip install "huggingface_hub[cli]"
huggingface-cli download DAMOXR/UCM --local-dir ./workspace/pretrained/

Download models using modelscope-cli:

pip install modelscope
modelscope download --model DAMOXR/UCM --local_dir ./workspace/pretrained/
### Inference

Run inference code on our provided demo videos, which requires a GPU with ~42GB memory and ~5min to generate a 12s video (241 frames):

```bash
python main.py \
   --img_path examples/images/frame_0000.png \
   --traj_path examples/cameras/cameras_0000.json \
   --prompt "The video captures a serene and picturesque scene of a traditional Dutch village on a bright, sunny day. The sky is a vibrant blue with scattered white clouds, creating a perfect backdrop for the charming architecture and lush greenery. The camera pans slowly across the village, revealing a row of quaint houses with red-tiled roofs and brick facades, typical of Dutch design. Some houses have green-painted wooden shutters and doors, adding a touch of color to the scene.  A narrow cobblestone street runs through the village, lined with parked cars on both sides, indicating a peaceful residential area."

To obtain all demo videos, you can use the following instruction:

python main.py --metafile examples/examples.csv

Parameters

• --img_path: Path to your reference image.
• --traj_path: Path to your specific camera trajectory file (.json).
• --prompt: Text prompt.
• --save_folder: Path to your folder for saving generated videos.
• --camera_scale_factor: Scales the camera center within the trajectory to match the scale of the 3D scene representation.
• --num_denoising_steps: The number of denoising iterations. 20 reaches a balance for the Gradio demo. 50 is used in our paper.
• --guidance_scale: Classifier-Free Guidance scale. The default value 5.0 is recommended.
• --seed: Seed for initializing the random number generator, controlling the randomness of Gaussian noise sampling.
• --duration: Only the first Duration frames of the camera trajectory will be processed. -1 represents the whole trajectory.

🤖 Gradio Demo

• Local demo:

  gradio app.py
  

🤝 Contributing

• Welcome to open issues and pull requests.
• Welcome to optimize the inference speed and memory usage, e.g., through model quantization, distillation, or other acceleration techniques.

❤️ Acknowledgement

We have used codes from other great research work, including STream3R and Wan2.1. We sincerely thank the authors for their awesome works!

📜 Citation

If you find this work helpful, please consider citing:

@article{xu2026ucm,
  title={UCM: Unified Modeling of Camera Control and Memory with Time-aware Positional Encoding Warping for World Models},
  author={Xu, Tianxing and Wang, Zixuan and Wang, Guangyuan and Hu, Li and Zhang, Zhongyi and Zhang, Peng and Zhang, Bang and Zhang, Song-Hai},
  journal={arXiv preprint arXiv:2602.22960},
  year={2026}
}
@article{xu2026ucm,
  title={Ucm: Unifying camera control and memory with time-aware positional encoding warping for world models},
  author={Xu, Tianxing and Wang, Zixuan and Wang, Guangyuan and Hu, Li and Zhang, Zhongyi and Zhang, Peng and Zhang, Bang and Zhang, Song-Hai},
  journal={arXiv preprint arXiv:2602.22960},
  year={2026}
}