Update model card for RealCam-I2V

README.md

@@ -3,168 +3,159 @@ license: mit
 tags:
 - image-to-video
 - pytorch
+pipeline_tag: image-to-video
+library_name: diffusers
 ---
-# CamI2V: Camera-Controlled Image-to-Video Diffusion Model
+
+# RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control
 
 <div align="center">
-    <a href="https://arxiv.org/abs/2410.15957">
-        <img src="https://img.shields.io/static/v1?label=arXiv&message=2410.15957&color=b21d1a" style="display: inline-block; vertical-align: middle;">
-    </a>
-    <a href="https://zgctroy.github.io/CamI2V">
-        <img src="https://img.shields.io/static/v1?label=Project&message=Page&color=green" style="display: inline-block; vertical-align: middle;">
-    </a>
-    <a href="https://huggingface.co/MuteApo/CamI2V/tree/main">
-        <img src="https://img.shields.io/static/v1?label=HuggingFace&message=Checkpoints&color=blue" style="display: inline-block; vertical-align: middle;">
-    </a>
-</div> 
+    <a href="https://huggingface.co/papers/2502.10059"><img src="https://img.shields.io/static/v1?label=arXiv&message=2502.10059&color=b21b1b"></a>
+    <a href="https://zgctroy.github.io/RealCam-I2V"><img src="https://img.shields.io/static/v1?label=Project&message=Page&color=green"></a>
+    <a href="https://github.com/ZGCTroy/RealCam-I2V"><img src="https://img.shields.io/static/v1?label=GitHub&message=Code&color=blue"></a>
+    <a href="https://huggingface.co/MuteApo/RealCam-I2V"><img src="https://img.shields.io/static/v1?label=HuggingFace&message=Model&color=orange"></a>
+</div>
+
+## Abstract
+Recent advancements in camera-trajectory-guided image-to-video generation offer higher precision and better support for complex camera control compared to text-based approaches. However, they also introduce significant usability challenges, as users often struggle to provide precise camera parameters when working with arbitrary real-world images without knowledge of their depth nor scene scale. To address these real-world application issues, we propose RealCam-I2V, a novel diffusion-based video generation framework that integrates monocular metric depth estimation to establish 3D scene reconstruction in a preprocessing step. During training, the reconstructed 3D scene enables scaling camera parameters from relative to metric scales, ensuring compatibility and scale consistency across diverse real-world images. In inference, RealCam-I2V offers an intuitive interface where users can precisely draw camera trajectories by dragging within the 3D scene. To further enhance precise camera control and scene consistency, we propose scene-constrained noise shaping, which shapes high-level noise and also allows the framework to maintain dynamic and coherent video generation in lower noise stages. RealCam-I2V achieves significant improvements in controllability and video quality on the RealEstate10K and out-of-domain images. We further enables applications like camera-controlled looping video generation and generative frame interpolation.
 
 ## 🎥 Gallery
+<div align="center">
+    <a href='https://zgctroy.github.io/RealCam-I2V'><img src="https://zgctroy.github.io/RealCam-I2V/assets/demo.gif" alt="RealCam-I2V Demo GIF" style="width: 100%; max-width: 650px;"></a>
+</div>
 
-<table>
-    <tr>
-        <td align="center">
-            rightward rotation and zoom in<br>(CFG=4, FS=6, step=50, ratio=0.6, scale=0.1)
-        </td>
-        <td align="center">
-            leftward rotation and zoom in<br>(CFG=4, FS=6, step=50, ratio=0.6, scale=0.1)
-        </td>
-    </tr>
-    <tr>
-        <td align="center">
-            <img src="https://github.com/user-attachments/assets/74a764f4-0631-4fbe-94b9-af51057f99a5" width="75%">
-        </td>
-        <td align="center">
-            <img src="https://github.com/user-attachments/assets/99309759-8355-4ee1-95c4-897f01c46720" width="75%">
-        </td>
-    </tr>
-    <tr>
-        <td align="center">
-            zoom in and upward movement<br>(CFG=4, FS=6, step=50, ratio=0.8, scale=0.2)
-        </td>
-        <td align="center">
-            downward movement and zoom-out<br>(CFG=4, FS=6, step=50, ratio=0.8, scale=0.2)
-        </td>
-    </tr>
-    <tr>
-        <td align="center">
-            <img src="https://github.com/user-attachments/assets/aef4cc2e-fd7e-46db-82bc-a7e59aab5963" width="75%">
-        </td>
-        <td align="center">
-            <img src="https://github.com/user-attachments/assets/f204992a-d729-492c-a663-85f9b80680f5" width="75%">
-        </td>
-    </tr>
-</table>
-
-## 🌟 News and Todo List
-
-- 🔥 25/03/17: Upload test metadata used in our paper to make easier evaluation.
-- 🔥 25/02/15: Release demo of [RealCam-I2V](https://zgctroy.github.io/RealCam-I2V/) for real-world applications, code will be available at [repo](https://github.com/ZGCTroy/RealCam-I2V).
-- 🔥 25/01/12: Release checkpoint of [CamI2V (512x320, 100k)](https://huggingface.co/MuteApo/CamI2V/blob/main/512_cami2v_100k.pt). We plan to release a more advanced model with longer training soon.
-- 🔥 25/01/02: Release checkpoint of [CamI2V (512x320, 50k)](https://huggingface.co/MuteApo/CamI2V/blob/main/512_cami2v_50k.pt), which is suitable for research propose and comparison.
-- 🔥 24/12/24: Integrate [Qwen2-VL](https://github.com/QwenLM/Qwen2-VL) in gradio demo, you can now caption your own input image by this powerful VLM.
-- 🔥 24/12/23: Release checkpoint of [CamI2V (256x256, 50k)](https://huggingface.co/MuteApo/CamI2V/blob/main/256_cami2v.pt).
-- 🔥 24/12/16: Release reproduced non-official checkpoints of [MotionCtrl (256x256, 50k)](https://huggingface.co/MuteApo/CamI2V/blob/main/256_motionctrl.pt) and [CameraCtrl (256x256, 50k)](https://huggingface.co/MuteApo/CamI2V/blob/main/256_cameractrl.pt) on [DynamiCrafter](https://github.com/Doubiiu/DynamiCrafter).
-- 🔥 24/12/09: Release training configs and scripts.
-- 🔥 24/12/06: Release [dataset pre-process code](datasets) for RealEstate10K.
-- 🔥 24/12/02: Release [evaluation code](evaluation) for RotErr, TransErr, CamMC and FVD.
-- 🌱 24/11/16: Release model code of CamI2V for training and inference, including implementation for MotionCtrl and CameraCtrl.
-
-## 📈 Performance
-
-Measured under 256x256 resolution, 50k training steps, 25 DDIM steps, text-image CFG 7.5, camera CFG 1.0 (no camera CFG).
-
-| Method        |  RotErr↓   | TransErr↓  |   CamMC↓   | FVD↓<br>(VideoGPT) | FVD↓<br>(StyleGAN) |
-| :------------ | :--------: | :--------: | :--------: | :----------------: | :----------------: |
-| DynamiCrafter |   3.3415   |   9.8024   |   11.625   |       106.02       |       92.196       |
-| MotionCtrl    |   0.8636   |   2.5068   |   2.9536   |       70.820       |       60.363       |
-| CameraCtrl    |   0.7064   |   1.9379   |   2.3070   |       66.713       |       57.644       |
-| CamI2V        | **0.4120** | **1.3409** | **1.5291** |     **62.439**     |     **53.361**     |
-
-### Inference Speed and GPU Memory
-
-| Method        | # Parameters | GPU Memory | Generation Time<br>(RTX 3090) |
-| :------------ | :----------: | :--------: | :---------------------------: |
-| DynamiCrafter |    1.4 B     | 11.14 GiB  |            8.14 s             |
-| MotionCtrl    |   + 63.4 M   | 11.18 GiB  |            8.27 s             |
-| CameraCtrl    |   + 211 M    | 11.56 GiB  |            8.38 s             |
-| CamI2V        |   + 261 M    | 11.67 GiB  |            10.3 s             |
+## 🌟 News
+- **25/07/05**: Release inference code and checkpoints of RealCam-I2V. We are still actively working on sanitizing the code. More updates of code and checkpoint will follow soon, please stay tuned!
+- **25/06/26**: RealCam-I2V is accepted by ICCV 2025! 🎉🎉
+- **25/05/18**: Release training code of RealCam-I2V on CogVideoX 1.5.
+- **25/03/26**: Release our dataset [RealCam-Vid](https://huggingface.co/datasets/MuteApo/RealCam-Vid) v1 for metric-scale camera-controlled video generation!
+- **25/02/18**: Initial commit of the project, we plan to release our DiT-based real-camera i2v models (e.g., CogVideoX) in this repo.
 
 ## ⚙️ Environment
 
 ### Quick Start
 
 ```shell
-conda create -n cami2v python=3.10
-conda activate cami2v
-
-conda install -y pytorch==2.4.1 torchvision==0.19.1 pytorch-cuda=12.1 -c pytorch -c nvidia
-conda install -y xformers -c xformers
+apt install libgl1-mesa-glx libgl1-mesa-dri xvfb # for ubuntu
+yum install -y mesa-libGL mesa-dri-drivers Xvfb. # for centos
+conda install ffmpeg=7 -c conda-forge
 pip install -r requirements.txt
 ```
 
 ## 💫 Inference
 
-### Download Model Checkpoints
-
-| Model      | Resolution |                                                                    Training Steps                                                                    |
-| :--------- | :--------: | :--------------------------------------------------------------------------------------------------------------------------------------------------: |
-| CamI2V     |  512x320   | [50k](https://huggingface.co/MuteApo/CamI2V/blob/main/512_cami2v_50k.pt), [100k](https://huggingface.co/MuteApo/CamI2V/blob/main/512_cami2v_100k.pt) |
-| CamI2V     |  256x256   |                                         [50k](https://huggingface.co/MuteApo/CamI2V/blob/main/256_cami2v.pt)                                         |
-| CameraCtrl |  256x256   |                                       [50k](https://huggingface.co/MuteApo/CamI2V/blob/main/256_cameractrl.pt)                                       |
-| MotionCtrl |  256x256   |                                       [50k](https://huggingface.co/MuteApo/CamI2V/blob/main/256_motionctrl.pt)                                       |
-
-Currently we release 256x256 checkpoints with 50k training steps of DynamiCrafter-based CamI2V, CameraCtrl and MotionCtrl, which is suitable for research propose and comparison.
+### Download Pretrained Models
 
-We also release 512x320 checkpoints of our CamI2V with longer training, make possible higher resolution and more advanced camera-controlled video generation.
+Download and put under `pretrained` folder the pretrained weights of [CogVideoX1.5-5B-I2V](https://huggingface.co/THUDM/CogVideoX1.5-5B-I2V), [Metric3D](https://huggingface.co/JUGGHM/Metric3D) and [Qwen2.5-VL](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct).
 
-Download above checkpoints and put under `ckpts` folder.
-Please edit `ckpt_path` in `configs/models.json` if you have a different model path.
-
-### Download Qwen2-VL Captioner (Optional)
+### Download Model Checkpoints
 
-Not required but recommend.
-It is used to caption your custom image in gradio demo for video generaion.
-We prefer the [AWQ](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct-AWQ) quantized version of Qwen2-VL due to speed and GPU memory.
+Download our weights of [RealCam-I2V](https://huggingface.co/MuteApo/RealCam-I2V) and put under `checkpoints` folder.
+Please edit `demo/models.json` if you have a custom model path.
 
-Download the pre-trained model and put under `pretrained_models` folder:
+### Run Gradio Demo
 
 ```shell
-─┬─ pretrained_models/
- └─── Qwen2-VL-7B-Instruct-AWQ/
+python gradio_app.py
 ```
 
-### Run Gradio Demo
+### Inference Code Example
+
+```python
+from transformers import AutoModel, AutoProcessor
+from PIL import Image
+import torch
+import cv2
+import numpy as np
+
+# Load RealCam-I2V model and processor
+model_path = "MuteApo/RealCam-I2V" # Or your local path to the checkpoint
+model = AutoModel.from_pretrained(
+    model_path,
+    torch_dtype=torch.float16, # Use torch.float32 for full precision
+    trust_remote_code=True,
+)
+processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)
+
+# Move model to GPU
+model.to("cuda")
+
+# Prepare inputs
+input_image_path = "./path/to/your/image.jpg" # Replace with your image path
+input_image = Image.open(input_image_path).convert("RGB")
+
+# Example camera trajectory (adjust as needed for your desired motion)
+# This is a simplified example; full camera control involves more parameters
+# See project page or original repo for detailed camera trajectory specifications.
+# Here, a simple stationary camera for 16 frames as an illustration.
+camera_trajectory = {
+    "center": [(0, 0, 0) for _ in range(16)],  # (x, y, z) position
+    "look_at": [(0, 0, 1) for _ in range(16)], # (x, y, z) point camera looks at
+    "up": [(0, 1, 0) for _ in range(16)],      # (x, y, z) up vector
+    "fovy": [45.0 for _ in range(16)],         # Field of view in degrees
+}
 
-```shell
-python cami2v_gradio_app.py --use_qwenvl_captioner
+# Process inputs
+inputs = processor(
+    images=input_image,
+    camera_trajectory=camera_trajectory,
+    return_tensors="pt"
+)
+inputs = {k: v.to("cuda") for k, v in inputs.items()}
+
+# Generate video
+with torch.no_grad():
+    video_frames = model.generate(**inputs, num_inference_steps=50).cpu().numpy()
+
+# Save video frames as a GIF or MP4
+output_video_path = "./output_video.gif" # or .mp4
+# Assuming video_frames are in [B, C, H, W] range [0,1]
+# Convert to [B, H, W, C] and scale to [0, 255] for saving
+video_frames = (video_frames * 255).astype(np.uint8).transpose(0, 2, 3, 1)
+
+# Example to save as GIF using imageio
+from imageio import mimsave
+mimsave(output_video_path, video_frames, fps=8) # Adjust fps as needed
+
+print(f"Video saved to {output_video_path}")
 ```
 
-Gradio may struggle to establish network connection, please re-try with `--use_host_ip`.
+## 🚀 Training
 
-## 🤗 Related Repo
+### Prepare Dataset
+
+Please access [RealCam-Vid](https://github.com/ZGCTroy/RealCam-Vid) and download our dataset for training RealCam-I2V-CogVideoX-1.5. Please unzip all contents in `data` folder.
+
+### Launch
 
-[RealCam-I2V: https://github.com/ZGCTroy/RealCam-I2V](https://github.com/ZGCTroy/RealCam-I2V)
+Edit example training script `accelerate_train.sh` if necessary and launch training by:
 
-[CameraCtrl: https://github.com/hehao13/CameraCtrl](https://github.com/hehao13/CameraCtrl)
+```shell
+bash accelerate_train.sh
+```
 
-[MotionCtrl: https://github.com/TencentARC/MotionCtrl](https://github.com/TencentARC/MotionCtrl)
+## 🤗 Related Repo
 
-[DynamiCrafter: https://github.com/Doubiiu/DynamiCrafter](https://github.com/Doubiiu/DynamiCrafter)
+- Our dataset, the first open-sourced, combining diverse scene dynamics with metric-scale camera trajectories, is available at [RealCam-Vid](https://github.com/ZGCTroy/RealCam-Vid).
+- Our previous work at [CamI2V](https://github.com/ZGCTroy/CamI2V).
+- We have borrowed a lot of code from the original [CogVideoX](https://github.com/THUDM/CogVideo) repository.
 
 ## 🗒️ Citation
 
-```
-@article{zheng2024cami2v,
-  title={CamI2V: Camera-Controlled Image-to-Video Diffusion Model},
-  author={Zheng, Guangcong and Li, Teng and Jiang, Rui and Lu, Yehao and Wu, Tao and Li, Xi},
-  journal={arXiv preprint arXiv:2410.15957},
-  year={2024}
-}
+If you find this work useful, please consider citing our papers:
 
+```bibtex
 @article{li2025realcam,
     title={RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control}, 
     author={Li, Teng and Zheng, Guangcong and Jiang, Rui and Zhan, Shuigen and Wu, Tao and Lu, Yehao and Lin, Yining and Li, Xi},
     journal={arXiv preprint arXiv:2502.10059},
     year={2025},
 }
+
+@article{zheng2024cami2v,
+    title={CamI2V: Camera-Controlled Image-to-Video Diffusion Model},
+    author={Zheng, Guangcong and Li, Teng and Jiang, Rui and Lu, Yehao and Wu, Tao and Li, Xi},
+    journal={arXiv preprint arXiv:2410.15957},
+    year={2024}
+}
 ```