Upload 8 files

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Yixuan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README-copy.md

@@ -0,0 +1,308 @@
+# ReCamMaster: Camera-Controlled Generative Rendering from A Single Video (ICCV'25 Oral, Best Paper Finalist)
+
+<div align="center">
+<div align="center" style="margin-top: 0px; margin-bottom: 0px;">
+<img src=https://github.com/user-attachments/assets/81ccf80e-f4b6-4a3d-b47a-e9c2ce14e34f width="30%"/>
+</div>
+
+### [<a href="https://arxiv.org/abs/2503.11647" target="_blank">arXiv</a>] [<a href="https://jianhongbai.github.io/ReCamMaster/" target="_blank">Project Page</a>] [<a href="https://huggingface.co/datasets/KwaiVGI/MultiCamVideo-Dataset" target="_blank">Dataset</a>]
+_**[Jianhong Bai<sup>1*</sup>](https://jianhongbai.github.io/), [Menghan Xia<sup>2†</sup>](https://menghanxia.github.io/), [Xiao Fu<sup>3</sup>](https://fuxiao0719.github.io/), [Xintao Wang<sup>2</sup>](https://xinntao.github.io/), [Lianrui Mu<sup>1</sup>](https://scholar.google.com/citations?user=dCik-2YAAAAJ&hl=en), [Jinwen Cao<sup>2</sup>](https://openreview.net/profile?id=~Jinwen_Cao1), <br>[Zuozhu Liu<sup>1</sup>](https://person.zju.edu.cn/en/lzz), [Haoji Hu<sup>1†</sup>](https://person.zju.edu.cn/en/huhaoji), [Xiang Bai<sup>4</sup>](https://scholar.google.com/citations?user=UeltiQ4AAAAJ&hl=en), [Pengfei Wan<sup>2</sup>](https://scholar.google.com/citations?user=P6MraaYAAAAJ&hl=en), [Di Zhang<sup>2</sup>](https://openreview.net/profile?id=~Di_ZHANG3)**_
+<br>
+(*Work done during an internship at KwaiVGI, Kuaishou Technology †corresponding authors)
+
+<sup>1</sup>Zhejiang University, <sup>2</sup>Kuaishou Technology, <sup>3</sup>CUHK, <sup>4</sup>HUST.
+
+</div>
+
+**Important Note:** This open-source repository is intended to provide a reference implementation. Due to the difference in the underlying T2V model's performance, the open-source version may not achieve the same performance as the model in our paper. If you'd like to use the best version of ReCamMaster, please upload your video to [this link](https://docs.google.com/forms/d/e/1FAIpQLSezOzGPbm8JMXQDq6EINiDf6iXn7rV4ozj6KcbQCSAzE8Vsnw/viewform?usp=dialog). Additionally, we are working on developing an online trial website. Please stay tuned to updates on the [Kling website](https://app.klingai.com/global/).
+
+## 🔥 Updates
+- __[2025.04.15]__: Please feel free to explore our related work, [SynCamMaster](https://github.com/KwaiVGI/SynCamMaster).
+- __[2025.04.09]__: Release the [training and inference code](https://github.com/KwaiVGI/ReCamMaster?tab=readme-ov-file#%EF%B8%8F-code-recammaster--wan21-inference--training), [model checkpoint](https://huggingface.co/KwaiVGI/ReCamMaster-Wan2.1/blob/main/step20000.ckpt).
+- __[2025.03.31]__: Release the [MultiCamVideo Dataset](https://huggingface.co/datasets/KwaiVGI/MultiCamVideo-Dataset).
+- __[2025.03.31]__: We have sent the inference results to the first 1000 trial users.
+- __[2025.03.17]__: Release the [project page](https://jianhongbai.github.io/ReCamMaster/) and the [try out link](https://docs.google.com/forms/d/e/1FAIpQLSezOzGPbm8JMXQDq6EINiDf6iXn7rV4ozj6KcbQCSAzE8Vsnw/viewform?usp=dialog).
+
+
+
+  
+## 📖 Introduction
+
+**TL;DR:** We propose ReCamMaster to re-capture in-the-wild videos with novel camera trajectories, achieved through our proposed simple-and-effective video conditioning scheme. We also release a multi-camera synchronized video [dataset](https://huggingface.co/datasets/KwaiVGI/MultiCamVideo-Dataset) rendered with Unreal Engine 5. <br>
+
+https://github.com/user-attachments/assets/52455e86-1adb-458d-bc37-4540a65a60d4
+
+## 🚀 Trail: Try ReCamMaster with Your Own Videos
+
+**Update:** We are actively processing the videos uploaded by users. So far, we have sent the inference results to the email addresses of the first **1500** testers. You should receive an email titled "Inference Results of ReCamMaster" from either jianhongbai@zju.edu.cn or cpurgicn@gmail.com. Please also check your spam folder, and let us know if you haven't received the email after a long time. If you enjoyed the videos we created, please consider giving us a star 🌟.
+
+**You can try out our ReCamMaster by uploading your own video to [this link](https://docs.google.com/forms/d/e/1FAIpQLSezOzGPbm8JMXQDq6EINiDf6iXn7rV4ozj6KcbQCSAzE8Vsnw/viewform?usp=dialog), which will generate a video with camera movements along a new trajectory.** We will send the mp4 file generated by ReCamMaster to your inbox as soon as possible. For camera movement trajectories, we offer 10 basic camera trajectories as follows:
+
+| Index       | Basic Trajectory                  |
+|-------------------|-----------------------------|
+| 1    | Pan Right                   |
+| 2 | Pan Left                    |
+| 3 | Tilt Up                     |
+| 4 | Tilt Down                   |
+| 5 | Zoom In                     |
+| 6 | Zoom Out                    |
+| 7 | Translate Up (with rotation)   |
+| 8 | Translate Down (with rotation) |
+| 9 | Arc Left (with rotation)    |
+| 10 | Arc Right (with rotation)   |
+
+If you would like to use ReCamMaster as a baseline and need qualitative or quantitative comparisons, please feel free to drop an email to [jianhongbai@zju.edu.cn](mailto:jianhongbai@zju.edu.cn). We can assist you with batch inference of our model.
+
+## ⚙️ Code: ReCamMaster + Wan2.1 (Inference & Training)
+The model utilized in our paper is an internally developed T2V model, not [Wan2.1](https://github.com/Wan-Video/Wan2.1). Due to company policy restrictions, we are unable to open-source the model used in the paper. Consequently, we migrated ReCamMaster to Wan2.1 to validate the effectiveness of our method. Due to differences in the underlying T2V model, you may not achieve the same results as demonstrated in the demo.
+### Inference
+Step 1: Set up the environment
+
+[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) requires Rust and Cargo to compile extensions. You can install them using the following command:
+```shell
+curl --proto '=https' --tlsv1.2 -sSf [https://sh.rustup.rs](https://sh.rustup.rs/) | sh
+. "$HOME/.cargo/env"
+```
+
+Install [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio):
+```shell
+git clone https://github.com/KwaiVGI/ReCamMaster.git
+cd ReCamMaster
+pip install -e .
+```
+
+Step 2: Download the pretrained checkpoints
+1. Download the pre-trained Wan2.1 models
+
+```shell
+cd ReCamMaster
+python download_wan2.1.py
+```
+2. Download the pre-trained ReCamMaster checkpoint
+
+Please download from [huggingface](https://huggingface.co/KwaiVGI/ReCamMaster-Wan2.1/blob/main/step20000.ckpt) and place it in ```models/ReCamMaster/checkpoints```.
+
+Step 3: Test the example videos
+```shell
+python inference_recammaster.py --cam_type 1
+```
+
+Step 4: Test your own videos
+
+If you want to test your own videos, you need to prepare your test data following the structure of the ```example_test_data``` folder. This includes N mp4 videos, each with at least 81 frames, and a ```metadata.csv``` file that stores their paths and corresponding captions. You can refer to the [Prompt Extension section](https://github.com/Wan-Video/Wan2.1?tab=readme-ov-file#2-using-prompt-extension) in Wan2.1 for guidance on preparing video captions. 
+
+```shell
+python inference_recammaster.py --cam_type 1 --dataset_path path/to/your/data
+```
+
+We provide several preset camera types, as shown in the table below. Additionally, you can generate new trajectories for testing.
+
+| cam_type       | Trajectory                  |
+|-------------------|-----------------------------|
+| 1    | Pan Right                   |
+| 2 | Pan Left                    |
+| 3 | Tilt Up                     |
+| 4 | Tilt Down                   |
+| 5 | Zoom In                     |
+| 6 | Zoom Out                    |
+| 7 | Translate Up (with rotation)   |
+| 8 | Translate Down (with rotation) |
+| 9 | Arc Left (with rotation)    |
+| 10 | Arc Right (with rotation)   |
+
+### Training
+
+Step 1: Set up the environment
+
+```shell
+pip install lightning pandas websockets
+```
+
+Step 2: Prepare the training dataset
+
+1. Download the [MultiCamVideo dataset](https://huggingface.co/datasets/KwaiVGI/MultiCamVideo-Dataset).
+
+2. Extract VAE features
+
+```shell
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python train_recammaster.py   --task data_process   --dataset_path path/to/the/MultiCamVideo/Dataset   --output_path ./models   --text_encoder_path "models/Wan-AI/Wan2.1-T2V-1.3B/models_t5_umt5-xxl-enc-bf16.pth"   --vae_path "models/Wan-AI/Wan2.1-T2V-1.3B/Wan2.1_VAE.pth"   --tiled   --num_frames 81   --height 480   --width 832 --dataloader_num_workers 2
+```
+
+3. Generate Captions for Each Video
+
+You can use video caption tools like [LLaVA](https://github.com/haotian-liu/LLaVA) to generate captions for each video and store them in the ```metadata.csv``` file.
+
+Step 3: Training
+```shell
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python train_recammaster.py   --task train  --dataset_path recam_train_data   --output_path ./models/train   --dit_path "models/Wan-AI/Wan2.1-T2V-1.3B/diffusion_pytorch_model.safetensors"   --steps_per_epoch 8000   --max_epochs 100   --learning_rate 1e-4   --accumulate_grad_batches 1   --use_gradient_checkpointing  --dataloader_num_workers 4
+```
+We do not explore the optimal set of hyper-parameters and train with a batch size of 1 on each GPU. You may achieve better model performance by adjusting hyper-parameters such as the learning rate and increasing the batch size.
+
+Step 4: Test the model
+
+```shell
+python inference_recammaster.py --cam_type 1 --ckpt_path path/to/the/checkpoint
+```
+
+## 📷 Dataset: MultiCamVideo Dataset
+### 1. Dataset Introduction
+
+**TL;DR:** The MultiCamVideo Dataset is a multi-camera synchronized video dataset rendered using Unreal Engine 5. It includes synchronized multi-camera videos and their corresponding camera trajectories. The MultiCamVideo Dataset can be valuable in fields such as camera-controlled video generation, synchronized video production, and 3D/4D reconstruction. If you are looking for synchronized videos captured with stationary cameras, please explore our [SynCamVideo Dataset](https://github.com/KwaiVGI/SynCamMaster).
+
+https://github.com/user-attachments/assets/6fa25bcf-1136-43be-8110-b527638874d4
+
+The MultiCamVideo Dataset is a multi-camera synchronized video dataset rendered using Unreal Engine 5. It includes synchronized multi-camera videos and their corresponding camera trajectories.
+It consists of 13.6K different dynamic scenes, each captured by 10 cameras, resulting in a total of 136K videos. Each dynamic scene is composed of four elements: {3D environment, character, animation, camera}. Specifically, we use animation to drive the character, 
+and position the animated character within the 3D environment. Then, Time-synchronized cameras are set up to move along predefined trajectories to render the multi-camera video data.
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/107c9607-e99b-4493-b715-3e194fcb3933" alt="Example Image" width="70%">
+</p>
+
+**3D Environment:** We collect 37 high-quality 3D environments assets from [Fab](https://www.fab.com). To minimize the domain gap between rendered data and real-world videos, we primarily select visually realistic 3D scenes, while choosing a few stylized or surreal 3D scenes as a supplement. To ensure data diversity, the selected scenes cover a variety of indoor and outdoor settings, such as city streets, shopping malls, cafes, office rooms, and the countryside.
+
+**Character:** We collect 66 different human 3D models as characters from [Fab](https://www.fab.com) and [Mixamo](https://www.mixamo.com).
+
+**Animation:** We collect 93 different animations from [Fab](https://www.fab.com) and [Mixamo](https://www.mixamo.com), including common actions such as waving, dancing, and cheering. We use these animations to drive the collected characters and create diverse datasets through various combinations.
+
+**Camera:** To ensure camera movements are diverse and closely resemble real-world distributions, we create a wide range of camera trajectories and parameters to cover various situations. To achieve this by designing rules to batch-generate random camera starting positions and movement trajectories:
+
+1. Camera Starting Position.
+
+We take the character's position as the center of a hemisphere with a radius of {3m, 5m, 7m, 10m} based on the size of the 3D scene and randomly sample within this range as the camera's starting point, ensuring the closest distance to the character is greater than 0.5m and the pitch angle is within 45 degrees.
+
+2. Camera Trajectories.
+
+- **Pan & Tilt**:  
+  The camera rotation angles are randomly selected within the range, with pan angles ranging from 5 to 45 degrees and tilt angles ranging from 5 to 30 degrees, with directions randomly chosen left/right or up/down.
+
+- **Basic Translation**:  
+  The camera translates along the positive and negative directions of the xyz axes, with movement distances randomly selected within the range of $[\frac{1}{4}, 1] \times \text{distance2character}$.
+
+- **Basic Arc Trajectory**:  
+  The camera moves along an arc, with rotation angles randomly selected within the range of 15 to 75 degrees.
+
+- **Random Trajectories**:  
+  1-3 points are sampled in space, and the camera moves from the initial position through these points as the movement trajectory, with the total movement distance randomly selected within the range of $[\frac{1}{4}, 1] \times \text{distance2character}$. The polyline is smoothed to make the movement more natural.
+
+- **Static Camera**:  
+  The camera does not translate or rotate during shooting, maintaining a fixed position.
+
+3. Camera Movement Speed.
+
+To further enhance the diversity of trajectories, 50% of the training data uses constant-speed camera trajectories, while the other 50% uses variable-speed trajectories generated by nonlinear functions. Consider a camera trajectory with a total of $f$ frames, starting at location $L_{start}$ and ending at position $L_{end}$. The location at the $i$-th frame is given by:
+```math
+L_i = L_{start} + (L_{end} - L_{start}) \cdot \left( \frac{1 - \exp(-a \cdot i/f)}{1 - \exp(-a)} \right),
+```
+where $a$ is an adjustable parameter to control the trajectory speed. When $a > 0$, the trajectory starts fast and then slows down; when $a < 0$, the trajectory starts slow and then speeds up. The larger the absolute value of $a$, the more drastic the change.
+
+4. Camera Parameters.
+
+We chose four set of camera parameters: {focal=18mm, aperture=10}, {focal=24mm, aperture=5}, {focal=35mm, aperture=2.4} and {focal=50mm, aperture=2.4}.
+
+### 2. Statistics and Configurations
+
+Dataset Statistics:
+
+| Number of Dynamic Scenes | Camera per Scene | Total Videos |
+|:------------------------:|:----------------:|:------------:|
+| 13,600                   | 10               | 136,000      |
+
+Video Configurations:
+
+| Resolution  | Frame Number | FPS                      |
+|:-----------:|:------------:|:------------------------:|
+| 1280x1280   | 81           | 15                       |
+
+Note: You can use 'center crop' to adjust the video's aspect ratio to fit your video generation model, such as 16:9, 9:16, 4:3, or 3:4.
+
+Camera Configurations:
+
+| Focal Length            | Aperture           | Sensor Height | Sensor Width |
+|:-----------------------:|:------------------:|:-------------:|:------------:|
+| 18mm, 24mm, 35mm, 50mm  | 10.0, 5.0, 2.4     | 23.76mm       | 23.76mm      |
+
+
+
+### 3. File Structure
+```
+MultiCamVideo-Dataset
+├── train
+│   ├── f18_aperture10
+│   │   ├── scene1    # one dynamic scene
+│   │   │   ├── videos
+│   │   │   │   ├── cam01.mp4    # synchronized 81-frame videos at 1280x1280 resolution
+│   │   │   │   ├── cam02.mp4
+│   │   │   │   ├── ...
+│   │   │   │   └── cam10.mp4
+│   │   │   └── cameras
+│   │   │       └── camera_extrinsics.json    # 81-frame camera extrinsics of the 10 cameras 
+│   │   ├── ...
+│   │   └── scene3400
+│   ├── f24_aperture5
+│   │   ├── scene1
+│   │   ├── ...
+│   │   └── scene3400
+│   ├── f35_aperture2.4
+│   │   ├── scene1
+│   │   ├── ...
+│   │   └── scene3400
+│   └── f50_aperture2.4
+│       ├── scene1
+│       ├── ...
+│       └── scene3400
+└── val
+    └── 10basic_trajectories
+        ├── videos
+        │   ├── cam01.mp4    # example videos corresponding to the validation cameras
+        │   ├── cam02.mp4
+        │   ├── ...
+        │   └── cam10.mp4
+        └── cameras
+            └── camera_extrinsics.json    # 10 different trajectories for validation
+```
+
+### 3. Useful scripts
+- Data Extraction
+```bash
+cat MultiCamVideo-Dataset.part* > MultiCamVideo-Dataset.tar.gz
+tar -xzvf MultiCamVideo-Dataset.tar.gz
+```
+- Camera Visualization
+```python
+python vis_cam.py
+```
+
+The visualization script is modified from [CameraCtrl](https://github.com/hehao13/CameraCtrl/blob/main/tools/visualize_trajectory.py), thanks for their inspiring work.
+
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/f9cf342d-2fb3-40ef-a7be-edafb5775004" alt="Example Image" width="40%">
+</p>
+
+## 🤗 Awesome Related Works
+Feel free to explore these outstanding related works, including but not limited to:
+
+[GCD](https://gcd.cs.columbia.edu/): GCD synthesizes large-angle novel viewpoints of 4D dynamic scenes from a monocular video.
+
+[ReCapture](https://generative-video-camera-controls.github.io/): a method for generating new videos with novel camera trajectories from a single user-provided video.
+
+[Trajectory Attention](https://xizaoqu.github.io/trajattn/): Trajectory Attention facilitates various tasks like camera motion control on images and videos, and video editing.
+
+[GS-DiT](https://wkbian.github.io/Projects/GS-DiT/): GS-DiT provides 4D video control for a single monocular video.
+
+[Diffusion as Shader](https://igl-hkust.github.io/das/): a versatile video generation control model for various tasks.
+
+[TrajectoryCrafter](https://trajectorycrafter.github.io/): TrajectoryCrafter achieves high-fidelity novel views generation from casually captured monocular video.
+
+[GEN3C](https://research.nvidia.com/labs/toronto-ai/GEN3C/): a generative video model with precise Camera Control and temporal 3D Consistency.
+
+## 🌟 Citation
+
+Please leave us a star 🌟 and cite our paper if you find our work helpful.
+```
+@article{bai2025recammaster,
+  title={ReCamMaster: Camera-Controlled Generative Rendering from A Single Video},
+  author={Bai, Jianhong and Xia, Menghan and Fu, Xiao and Wang, Xintao and Mu, Lianrui and Cao, Jinwen and Liu, Zuozhu and Hu, Haoji and Bai, Xiang and Wan, Pengfei and others},
+  journal={arXiv preprint arXiv:2503.11647},
+  year={2025}
+}
+```

README.md

@@ -1,3 +1,368 @@
----
-license: mit
----
+# Astra<img src="./assets/images/logo.png" alt="logo" style="height: 1em; vertical-align: baseline; margin: 0 0.1em;">: General Interactive World Model with Autoregressive Denoising
+
+<div align="center">
+<div align="center" style="margin-top: 0px; margin-bottom: -30px;">
+<img src="./assets/images/logo-text.png" width="30%"/>
+</div>
+
+### [<a href="https://arxiv.org/abs/2503.11647" target="_blank">arXiv</a>] [<a href="https://eternalevan.github.io/Astra-project/" target="_blank">Project Page</a>]
+**[Yixuan Zhu<sup>1</sup>](https://jianhongbai.github.io/), [Jiaqi Feng<sup>1</sup>](https://menghanxia.github.io/), [Wenzhao Zheng<sup>1 †</sup>](https://fuxiao0719.github.io/), [Yuan Gao<sup>2</sup>](https://xinntao.github.io/), [Xin Tao<sup>2</sup>](https://scholar.google.com/citations?user=dCik-2YAAAAJ&hl=en), [Pengfei Wan<sup>2</sup>](https://openreview.net/profile?id=~Jinwen_Cao1), [Jie Zhou <sup>1</sup>](https://person.zju.edu.cn/en/lzz), [Jiwen Lu<sup>1</sup>](https://person.zju.edu.cn/en/huhaoji)**
+<!-- <br> -->
+(*Work done during an internship at Kuaishou Technology, 
+† Project leader)
+
+<sup>1</sup>Tsinghua University, <sup>2</sup>Kuaishou Technology. 
+</div>
+
+## 🔥 Updates
+- __[2025.11.17]__: Release the [project page](https://eternalevan.github.io/Astra-project/).
+- __[2025.12.09]__: Release the training and inference code, model checkpoint.
+
+## 🎯 TODO List
+
+- [ ] **Release full inference pipelines** for additional scenarios:
+  - [ ] 🚗 Autonomous driving
+  - [ ] 🤖 Robotic manipulation
+  - [ ] 🛸 Drone navigation / exploration
+
+
+- [ ] **Open-source training scripts**:
+  - [ ] ⬆️ Action-conditioned autoregressive denoising training
+  - [ ] 🔄 Multi-scenario joint training pipeline
+
+- [ ] **Release dataset preprocessing tools**
+
+- [ ] **Provide unified evaluation toolkit**
+## 📖 Introduction
+
+**TL;DR:** Astra is an **interactive world model** that delivers realistic long-horizon video rollouts under a wide range of scenarios and action inputs.
+
+## Gallery
+
+### Astra+Wan2.1
+
+<table border="0" style="width: 100%; text-align: left; margin-top: 20px;">
+  <tr>
+      <td>
+          <video src="https://github.com/user-attachments/assets/715a5b66-3966-4923-aa00-02315fb07761"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+      </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/1451947e-1851-4b57-a666-a44ffea7b10c"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+      </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/c7156c4d-d51d-493c-995e-5113c3d49abb"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+     </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/f7550916-e224-497a-b0b9-84479607c962"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+     </td>
+  </tr>
+
+  <tr>
+      <td>
+          <video src="https://github.com/user-attachments/assets/d899d704-c706-4e64-a24b-eea174d2173d"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+      </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/c1d8beb2-3102-468a-8019-624d89fba125"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+      </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/2aabc10b-f945-4d9d-b24a-baed17fcfe14"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+     </td>
+      <td>
+          <video src="https://github.com/user-attachments/assets/5c03e6ae-0fc2-4e09-a5b5-f37d04e7bbf8"
+                 style="width:100%; height:180px; object-fit:cover;" 
+                 controls autoplay loop muted></video>
+     </td>
+  </tr>
+</table>
+
+<!-- ## 🚀 Trail: Try ReCamMaster with Your Own Videos
+
+**Update:** We are actively processing the videos uploaded by users. So far, we have sent the inference results to the email addresses of the first **1180** testers. You should receive an email titled "Inference Results of ReCamMaster" from either jianhongbai@zju.edu.cn or cpurgicn@gmail.com. Please also check your spam folder, and let us know if you haven't received the email after a long time. If you enjoyed the videos we created, please consider giving us a star 🌟.
+
+**You can try out our ReCamMaster by uploading your own video to [this link](https://docs.google.com/forms/d/e/1FAIpQLSezOzGPbm8JMXQDq6EINiDf6iXn7rV4ozj6KcbQCSAzE8Vsnw/viewform?usp=dialog), which will generate a video with camera movements along a new trajectory.** We will send the mp4 file generated by ReCamMaster to your inbox as soon as possible. For camera movement trajectories, we offer 10 basic camera trajectories as follows:
+
+| Index       | Basic Trajectory                  |
+|-------------------|-----------------------------|
+| 1    | Pan Right                   |
+| 2 | Pan Left                    |
+| 3 | Tilt Up                     |
+| 4 | Tilt Down                   |
+| 5 | Zoom In                     |
+| 6 | Zoom Out                    |
+| 7 | Translate Up (with rotation)   |
+| 8 | Translate Down (with rotation) |
+| 9 | Arc Left (with rotation)    |
+| 10 | Arc Right (with rotation)   |
+
+If you would like to use ReCamMaster as a baseline and need qualitative or quantitative comparisons, please feel free to drop an email to [jianhongbai@zju.edu.cn](mailto:jianhongbai@zju.edu.cn). We can assist you with batch inference of our model. -->
+
+## ⚙️ Code: Astra + Wan2.1 (Inference & Training)
+Astra is built upon [Wan2.1-1.3B](https://github.com/Wan-Video/Wan2.1), a diffusion-based video generation model. We provide inference scripts to help you quickly generate videos from images and action inputs. Follow the steps below:
+
+### Inference
+Step 1: Set up the environment
+
+[DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio) requires Rust and Cargo to compile extensions. You can install them using the following command:
+```shell
+curl --proto '=https' --tlsv1.2 -sSf [https://sh.rustup.rs](https://sh.rustup.rs/) | sh
+. "$HOME/.cargo/env"
+```
+
+Install [DiffSynth-Studio](https://github.com/modelscope/DiffSynth-Studio):
+```shell
+git clone https://github.com/EternalEvan/Astra.git
+cd Astra
+pip install -e .
+```
+
+Step 2: Download the pretrained checkpoints
+1. Download the pre-trained Wan2.1 models
+
+```shell
+cd script
+python download_wan2.1.py
+```
+2. Download the pre-trained Astra checkpoint
+
+Please download from [huggingface](https://huggingface.co/wjque/lyra/blob/main/diffusion_pytorch_model.ckpt) and place it in ```models/Astra/checkpoints```.
+
+Step 3: Test the example videos
+```shell
+python inference_astra.py --cam_type 1
+```
+
+Step 4: Test your own videos
+
+If you want to test your own videos, you need to prepare your test data following the structure of the ```example_test_data``` folder. This includes N mp4 videos, each with at least 81 frames, and a ```metadata.csv``` file that stores their paths and corresponding captions. You can refer to the [Prompt Extension section](https://github.com/Wan-Video/Wan2.1?tab=readme-ov-file#2-using-prompt-extension) in Wan2.1 for guidance on preparing video captions. 
+
+```shell
+python inference_astra.py --cam_type 1 --dataset_path path/to/your/data
+```
+
+We provide several preset camera types, as shown in the table below. Additionally, you can generate new trajectories for testing.
+
+| cam_type       | Trajectory                  |
+|-------------------|-----------------------------|
+| 1    | Pan Right                   |
+| 2 | Pan Left                    |
+| 3 | Tilt Up                     |
+| 4 | Tilt Down                   |
+| 5 | Zoom In                     |
+| 6 | Zoom Out                    |
+| 7 | Translate Up (with rotation)   |
+| 8 | Translate Down (with rotation) |
+| 9 | Arc Left (with rotation)    |
+| 10 | Arc Right (with rotation)   |
+
+### Training
+
+Step 1: Set up the environment
+
+```shell
+pip install lightning pandas websockets
+```
+
+Step 2: Prepare the training dataset
+
+1. Download the [MultiCamVideo dataset](https://huggingface.co/datasets/KwaiVGI/MultiCamVideo-Dataset).
+
+2. Extract VAE features
+
+```shell
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python train_recammaster.py   --task data_process   --dataset_path path/to/the/MultiCamVideo/Dataset   --output_path ./models   --text_encoder_path "models/Wan-AI/Wan2.1-T2V-1.3B/models_t5_umt5-xxl-enc-bf16.pth"   --vae_path "models/Wan-AI/Wan2.1-T2V-1.3B/Wan2.1_VAE.pth"   --tiled   --num_frames 81   --height 480   --width 832 --dataloader_num_workers 2
+```
+
+3. Generate Captions for Each Video
+
+You can use video caption tools like [LLaVA](https://github.com/haotian-liu/LLaVA) to generate captions for each video and store them in the ```metadata.csv``` file.
+
+Step 3: Training
+```shell
+CUDA_VISIBLE_DEVICES="0,1,2,3,4,5,6,7" python train_recammaster.py   --task train  --dataset_path recam_train_data   --output_path ./models/train   --dit_path "models/Wan-AI/Wan2.1-T2V-1.3B/diffusion_pytorch_model.safetensors"   --steps_per_epoch 8000   --max_epochs 100   --learning_rate 1e-4   --accumulate_grad_batches 1   --use_gradient_checkpointing  --dataloader_num_workers 4
+```
+We do not explore the optimal set of hyper-parameters and train with a batch size of 1 on each GPU. You may achieve better model performance by adjusting hyper-parameters such as the learning rate and increasing the batch size.
+
+Step 4: Test the model
+
+```shell
+python inference_recammaster.py --cam_type 1 --ckpt_path path/to/the/checkpoint
+```
+
+<!-- ## 📷 Dataset: MultiCamVideo Dataset
+### 1. Dataset Introduction
+
+**TL;DR:** The MultiCamVideo Dataset is a multi-camera synchronized video dataset rendered using Unreal Engine 5. It includes synchronized multi-camera videos and their corresponding camera trajectories. The MultiCamVideo Dataset can be valuable in fields such as camera-controlled video generation, synchronized video production, and 3D/4D reconstruction.
+
+https://github.com/user-attachments/assets/6fa25bcf-1136-43be-8110-b527638874d4
+
+The MultiCamVideo Dataset is a multi-camera synchronized video dataset rendered using Unreal Engine 5. It includes synchronized multi-camera videos and their corresponding camera trajectories.
+It consists of 13.6K different dynamic scenes, each captured by 10 cameras, resulting in a total of 136K videos. Each dynamic scene is composed of four elements: {3D environment, character, animation, camera}. Specifically, we use animation to drive the character, 
+and position the animated character within the 3D environment. Then, Time-synchronized cameras are set up to move along predefined trajectories to render the multi-camera video data.
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/107c9607-e99b-4493-b715-3e194fcb3933" alt="Example Image" width="70%">
+</p>
+
+**3D Environment:** We collect 37 high-quality 3D environments assets from [Fab](https://www.fab.com). To minimize the domain gap between rendered data and real-world videos, we primarily select visually realistic 3D scenes, while choosing a few stylized or surreal 3D scenes as a supplement. To ensure data diversity, the selected scenes cover a variety of indoor and outdoor settings, such as city streets, shopping malls, cafes, office rooms, and the countryside.
+
+**Character:** We collect 66 different human 3D models as characters from [Fab](https://www.fab.com) and [Mixamo](https://www.mixamo.com).
+
+**Animation:** We collect 93 different animations from [Fab](https://www.fab.com) and [Mixamo](https://www.mixamo.com), including common actions such as waving, dancing, and cheering. We use these animations to drive the collected characters and create diverse datasets through various combinations.
+
+**Camera:** To ensure camera movements are diverse and closely resemble real-world distributions, we create a wide range of camera trajectories and parameters to cover various situations. To achieve this by designing rules to batch-generate random camera starting positions and movement trajectories:
+
+1. Camera Starting Position.
+
+We take the character's position as the center of a hemisphere with a radius of {3m, 5m, 7m, 10m} based on the size of the 3D scene and randomly sample within this range as the camera's starting point, ensuring the closest distance to the character is greater than 0.5m and the pitch angle is within 45 degrees.
+
+2. Camera Trajectories.
+
+- **Pan & Tilt**:  
+  The camera rotation angles are randomly selected within the range, with pan angles ranging from 5 to 45 degrees and tilt angles ranging from 5 to 30 degrees, with directions randomly chosen left/right or up/down.
+
+- **Basic Translation**:  
+  The camera translates along the positive and negative directions of the xyz axes, with movement distances randomly selected within the range of $[\frac{1}{4}, 1] \times \text{distance2character}$.
+
+- **Basic Arc Trajectory**:  
+  The camera moves along an arc, with rotation angles randomly selected within the range of 15 to 75 degrees.
+
+- **Random Trajectories**:  
+  1-3 points are sampled in space, and the camera moves from the initial position through these points as the movement trajectory, with the total movement distance randomly selected within the range of $[\frac{1}{4}, 1] \times \text{distance2character}$. The polyline is smoothed to make the movement more natural.
+
+- **Static Camera**:  
+  The camera does not translate or rotate during shooting, maintaining a fixed position.
+
+3. Camera Movement Speed.
+
+To further enhance the diversity of trajectories, 50% of the training data uses constant-speed camera trajectories, while the other 50% uses variable-speed trajectories generated by nonlinear functions. Consider a camera trajectory with a total of $f$ frames, starting at location $L_{start}$ and ending at position $L_{end}$. The location at the $i$-th frame is given by:
+```math
+L_i = L_{start} + (L_{end} - L_{start}) \cdot \left( \frac{1 - \exp(-a \cdot i/f)}{1 - \exp(-a)} \right),
+```
+where $a$ is an adjustable parameter to control the trajectory speed. When $a > 0$, the trajectory starts fast and then slows down; when $a < 0$, the trajectory starts slow and then speeds up. The larger the absolute value of $a$, the more drastic the change.
+
+4. Camera Parameters.
+
+We chose four set of camera parameters: {focal=18mm, aperture=10}, {focal=24mm, aperture=5}, {focal=35mm, aperture=2.4} and {focal=50mm, aperture=2.4}.
+
+### 2. Statistics and Configurations
+
+Dataset Statistics:
+
+| Number of Dynamic Scenes | Camera per Scene | Total Videos |
+|:------------------------:|:----------------:|:------------:|
+| 13,600                   | 10               | 136,000      |
+
+Video Configurations:
+
+| Resolution  | Frame Number | FPS                      |
+|:-----------:|:------------:|:------------------------:|
+| 1280x1280   | 81           | 15                       |
+
+Note: You can use 'center crop' to adjust the video's aspect ratio to fit your video generation model, such as 16:9, 9:16, 4:3, or 3:4.
+
+Camera Configurations:
+
+| Focal Length            | Aperture           | Sensor Height | Sensor Width |
+|:-----------------------:|:------------------:|:-------------:|:------------:|
+| 18mm, 24mm, 35mm, 50mm  | 10.0, 5.0, 2.4     | 23.76mm       | 23.76mm      |
+
+
+
+### 3. File Structure
+```
+MultiCamVideo-Dataset
+├── train
+│   ├── f18_aperture10
+│   │   ├── scene1    # one dynamic scene
+│   │   │   ├── videos
+│   │   │   │   ├── cam01.mp4    # synchronized 81-frame videos at 1280x1280 resolution
+│   │   │   │   ├── cam02.mp4
+│   │   │   │   ├── ...
+│   │   │   │   └── cam10.mp4
+│   │   │   └── cameras
+│   │   │       └── camera_extrinsics.json    # 81-frame camera extrinsics of the 10 cameras 
+│   │   ├── ...
+│   │   └── scene3400
+│   ├── f24_aperture5
+│   │   ├── scene1
+│   │   ├── ...
+│   │   └── scene3400
+│   ├── f35_aperture2.4
+│   │   ├── scene1
+│   │   ├── ...
+│   │   └── scene3400
+│   └── f50_aperture2.4
+│       ├── scene1
+│       ├── ...
+│       └── scene3400
+└── val
+    └── 10basic_trajectories
+        ├── videos
+        │   ├── cam01.mp4    # example videos corresponding to the validation cameras
+        │   ├── cam02.mp4
+        │   ├── ...
+        │   └── cam10.mp4
+        └── cameras
+            └── camera_extrinsics.json    # 10 different trajectories for validation
+```
+
+### 3. Useful scripts
+- Data Extraction
+```bash
+cat MultiCamVideo-Dataset.part* > MultiCamVideo-Dataset.tar.gz
+tar -xzvf MultiCamVideo-Dataset.tar.gz
+```
+- Camera Visualization
+```python
+python vis_cam.py
+```
+
+The visualization script is modified from [CameraCtrl](https://github.com/hehao13/CameraCtrl/blob/main/tools/visualize_trajectory.py), thanks for their inspiring work.
+
+<p align="center">
+  <img src="https://github.com/user-attachments/assets/f9cf342d-2fb3-40ef-a7be-edafb5775004" alt="Example Image" width="40%">
+</p> -->
+
+## 🤗 Awesome Related Works
+Feel free to explore these outstanding related works, including but not limited to:
+
+[ReCamMaster](https://github.com/KlingTeam/ReCamMaster): ReCamMaster re-captures in-the-wild videos with novel camera trajectories. 
+
+[GCD](https://gcd.cs.columbia.edu/): GCD synthesizes large-angle novel viewpoints of 4D dynamic scenes from a monocular video.
+
+[ReCapture](https://generative-video-camera-controls.github.io/): a method for generating new videos with novel camera trajectories from a single user-provided video.
+
+[Trajectory Attention](https://xizaoqu.github.io/trajattn/): Trajectory Attention facilitates various tasks like camera motion control on images and videos, and video editing.
+
+[GS-DiT](https://wkbian.github.io/Projects/GS-DiT/): GS-DiT provides 4D video control for a single monocular video.
+
+[Diffusion as Shader](https://igl-hkust.github.io/das/): a versatile video generation control model for various tasks.
+
+[TrajectoryCrafter](https://trajectorycrafter.github.io/): TrajectoryCrafter achieves high-fidelity novel views generation from casually captured monocular video.
+
+[GEN3C](https://research.nvidia.com/labs/toronto-ai/GEN3C/): a generative video model with precise Camera Control and temporal 3D Consistency.
+
+## 🌟 Citation
+
+Please leave us a star 🌟 and cite our paper if you find our work helpful.
+```
+@inproceedings{zhu2025astra,
+  title={Astra: General Interactive World Model with Autoregressive Denoising},
+  author={Zhu, Yixuan and Feng, Jiaqi and Zheng, Wenzhao and Gao, Yuan and Tao, Xin and Wan, Pengfei and Zhou, Jie and Lu, Jiwen},
+  booktitle={arxiv},
+  year={2025}
+}
+```

Try ReCamMaster with Your Own Videos Here.txt

@@ -0,0 +1 @@
+https://docs.google.com/forms/d/e/1FAIpQLSezOzGPbm8JMXQDq6EINiDf6iXn7rV4ozj6KcbQCSAzE8Vsnw/viewform?usp=dialog

infer.sh

@@ -0,0 +1,56 @@
+CUDA_VISIBLE_DEVICES=0 python infer_origin.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/sekai-game-walking/00100100001_0004650_0004950/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/sekai.mp4 \
+    --prompt "A drone flying scene in a game world" \
+    --modality_type sekai
+
+CUDA_VISIBLE_DEVICES=1 python infer_moe.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/nuscenes_video_generation_dynamic/scenes/scene-0001_CAM_FRONT/encoded_video-480p.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/nuscenes.mp4 \
+    --prompt "A car is driving" \
+    --modality_type nuscenes
+
+CUDA_VISIBLE_DEVICES=0 python infer_origin.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/spatialvid/a9a6d37f-0a6c-548a-a494-7d902469f3f2_0000000_0000300/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/spatialvid.mp4 \
+    --prompt "A man is entering the room" \
+    --modality_type sekai
+
+CUDA_VISIBLE_DEVICES=1 python infer_moe.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/openx-fractal-encoded/episode_000001/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/openx.mp4 \
+    --prompt "A robotic arm is moving the object" \
+    --modality_type openx
+
+CUDA_VISIBLE_DEVICES=1 python infer_origin.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/sekai-game-drone/00500210001_0012150_0012450/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/edit.mp4 \
+    --prompt "A drone flying scene in a game world, and it starts to rain" \
+    --modality_type sekai
+
+
+CUDA_VISIBLE_DEVICES=0 python infer_origin.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/spatialvid/0268e6b0-f41e-5c2f-bf6b-936e55dc4a05_0000600_0000900/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/spatialvid.mp4 \
+    --prompt "walking in the city, the weather from day turns to night" \
+    --modality_type sekai \
+    --direction "right" \
+    --initial_condition_frames "1"
+
+    CUDA_VISIBLE_DEVICES=1 python infer_moe.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/nuscenes_video_generation_dynamic/scenes/scene-0001_CAM_FRONT/encoded_video-480p.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/nuscenes.mp4 \
+    --prompt "A car is driving" \
+    --modality_type nuscenes
+    
+    CUDA_VISIBLE_DEVICES=1 python infer_moe.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/openx-fractal-encoded/episode_000001/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/openx.mp4 \
+    --prompt "A robotic arm is moving the object" \
+    --modality_type openx
+
+    CUDA_VISIBLE_DEVICES=1 python infer_origin.py \
+    --condition_pth /share_zhuyixuan05/zhuyixuan05/sekai-game-walking/00100100001_0004650_0004950/encoded_video.pth \
+    --output_path /home/zhuyixuan05/ReCamMaster/moe/infer_results/sekai.mp4 \
+    --prompt "A drone flying scene in a game world" \
+    --modality_type sekai

infer_demo.sh

@@ -0,0 +1,12 @@
+CUDA_VISIBLE_DEVICES=3 python ./scripts/infer_demo.py \
+    --condition_pth ./examples/condition_pth/garden_1.pth \
+    --start_frame 0 \
+    --initial_condition_frames 1 \
+    --frames_per_generation 8 \
+    --total_frames_to_generate 24 \
+    --dit_path /path/to/your/model.pth \
+    --prompt "A sunlit European street lined with historic buildings and vibrant greenery creates a warm, charming, and inviting atmosphere. The scene shows a picturesque open square paved with red bricks, surrounded by classic narrow townhouses featuring tall windows, gabled roofs, and dark-painted facades. On the right side, a lush arrangement of potted plants and blooming flowers adds rich color and texture to the foreground. A vintage-style streetlamp stands prominently near the center-right, contributing to the timeless character of the street. Mature trees frame the background, their leaves glowing in the warm afternoon sunlight. Bicycles are visible along the edges of the buildings, reinforcing the urban yet leisurely feel. The sky is bright blue with scattered clouds, and soft sun flares enter the frame from the left, enhancing the scene’s inviting, peaceful mood." \
+    --add_icons \
+    --modality_type sekai \
+    --direction forward_left \
+    --output_path /mnt/data/louis_crq/astra2/astra_test/Astra/examples/output_videos/output_moe_framepack_sliding.mp4

pip-list.txt

@@ -0,0 +1,197 @@
+(ddrm) zyx@zwl:~$ pip  list
+Package                  Version         Editable project location
+------------------------ --------------- ---------------------------------------------------------
+absl-py                  2.0.0
+accelerate               1.0.1
+addict                   2.4.0
+aiohttp                  3.8.5
+aiosignal                1.3.1
+albumentations           1.4.6
+annotated-types          0.6.0
+antlr4-python3-runtime   4.9.3
+appdirs                  1.4.4
+asttokens                2.4.1
+async-timeout            4.0.3
+attrs                    23.1.0
+backcall                 0.2.0
+basicsr                  1.2.0+1.4.2     /home/zyx/Retinexformer-master
+beautifulsoup4           4.12.3
+blessed                  1.20.0
+blobfile                 2.0.2
+cachetools               5.3.1
+certifi                  2023.5.7
+cffi                     1.15.1
+charset-normalizer       3.2.0
+click                    8.1.7
+cmake                    3.26.4
+contourpy                1.1.1
+cycler                   0.11.0
+decorator                5.1.1
+decord                   0.6.0
+diffusers                0.31.0
+dlib                     19.24.6
+docker-pycreds           0.4.0
+einops                   0.6.1
+executing                2.1.0
+face-alignment           1.4.1
+facexlib                 0.3.0
+filelock                 3.12.2
+filterpy                 1.4.5
+fire                     0.5.0
+flatbuffers              23.5.26
+fonttools                4.42.1
+frozenlist               1.4.0
+fsspec                   2023.9.1
+ftfy                     6.1.1
+future                   0.18.3
+gdown                    5.2.0
+gfpgan                   1.3.8           /home/zyx/anaconda3/envs/ddrm/lib/python3.8/site-packages
+gitdb                    4.0.10
+GitPython                3.1.37
+google-auth              2.23.0
+google-auth-oauthlib     1.0.0
+gpustat                  1.1
+grpcio                   1.58.0
+guided-diffusion         0.0.0           /home/zyx/GenerativeDiffusionPrior
+huggingface-hub          0.30.2
+idna                     3.4
+imageio                  2.31.5
+imgaug                   0.4.0
+importlib-metadata       6.8.0
+importlib-resources      6.1.0
+ip-adapter               0.1.0
+ipython                  8.12.3
+jedi                     0.19.2
+Jinja2                   3.1.2
+joblib                   1.3.2
+kiwisolver               1.4.5
+lazy_loader              0.3
+lightning-utilities      0.9.0
+lit                      16.0.6
+llvmlite                 0.41.0
+lmdb                     1.4.1
+loguru                   0.7.2
+lora-diffusion           0.1.7
+loralib                  0.1.2
+lpips                    0.1.4
+lxml                     4.9.3
+Markdown                 3.4.4
+markdown-it-py           3.0.0
+MarkupSafe               2.1.3
+matplotlib               3.7.3
+matplotlib-inline        0.1.7
+mdurl                    0.1.2
+mediapipe                0.10.5
+mmcv                     1.7.0
+mmengine                 0.10.7
+mpi4py                   3.1.4
+mpmath                   1.3.0
+multidict                6.0.4
+mypy-extensions          1.0.0
+natsort                  8.4.0
+networkx                 3.1
+ninja                    1.11.1.1
+numba                    0.58.0
+numpy                    1.24.4
+nvidia-cublas-cu11       11.10.3.66
+nvidia-cuda-cupti-cu11   11.7.101
+nvidia-cuda-nvrtc-cu11   11.7.99
+nvidia-cuda-runtime-cu11 11.7.99
+nvidia-cudnn-cu11        8.5.0.96
+nvidia-cufft-cu11        10.9.0.58
+nvidia-curand-cu11       10.2.10.91
+nvidia-cusolver-cu11     11.4.0.1
+nvidia-cusparse-cu11     11.7.4.91
+nvidia-ml-py             12.535.77
+nvidia-nccl-cu11         2.14.3
+nvidia-nvtx-cu11         11.7.91
+oauthlib                 3.2.2
+omegaconf                2.3.0
+open-clip-torch          2.20.0
+openai-clip              1.0.1
+opencv-contrib-python    4.8.0.76
+opencv-python            4.8.0.74
+opencv-python-headless   4.9.0.80
+packaging                23.1
+pandas                   2.0.3
+parso                    0.8.4
+pathtools                0.1.2
+peft                     0.13.2
+pexpect                  4.9.0
+pickleshare              0.7.5
+Pillow                   10.0.0
+pip                      23.1.2
+platformdirs             3.11.0
+prompt_toolkit           3.0.48
+protobuf                 3.20.3
+psutil                   5.9.5
+ptyprocess               0.7.0
+pure_eval                0.2.3
+pyasn1                   0.5.0
+pyasn1-modules           0.3.0
+pycparser                2.21
+pycryptodomex            3.18.0
+pydantic                 2.7.1
+pydantic_core            2.18.2
+pyDeprecate              0.3.1
+Pygments                 2.18.0
+pyiqa                    0.1.8
+pyparsing                3.1.1
+pyre-extensions          0.0.23
+PySocks                  1.7.1
+python-dateutil          2.8.2
+pytorch-fid              0.3.0
+pytorch-lightning        1.4.2
+pytz                     2023.3.post1
+PyWavelets               1.4.1
+PyYAML                   6.0.1
+realesrgan               0.3.0
+regex                    2023.8.8
+requests                 2.31.0
+requests-oauthlib        1.3.1
+rich                     14.0.0
+rsa                      4.9
+safetensors              0.4.5
+scikit-image             0.21.0
+scikit-learn             1.3.2
+scipy                    1.10.1
+sentencepiece            0.1.99
+sentry-sdk               1.31.0
+setproctitle             1.3.2
+setuptools               68.0.0
+shapely                  2.0.1
+six                      1.16.0
+smmap                    5.0.1
+sounddevice              0.4.6
+soupsieve                2.5
+stack-data               0.6.3
+sympy                    1.12
+tb-nightly               2.14.0a20230808
+tensorboard              2.14.0
+tensorboard-data-server  0.7.1
+termcolor                2.3.0
+threadpoolctl            3.2.0
+tifffile                 2023.7.10
+timm                     0.9.7
+tokenizers               0.15.0
+tomli                    2.0.1
+torch                    1.13.1+cu116
+torchmetrics             0.5.0
+torchvision              0.14.1+cu116
+tqdm                     4.65.0
+traitlets                5.14.3
+transformers             4.36.2
+triton                   2.0.0
+typing_extensions        4.11.0
+typing-inspect           0.9.0
+tzdata                   2023.3
+urllib3                  1.26.16
+vqfr                     2.0.0           /home/zyx/VQFR
+wandb                    0.15.11
+wcwidth                  0.2.6
+Werkzeug                 2.3.7
+wheel                    0.40.0
+xformers                 0.0.16
+yapf                     0.40.2
+yarl                     1.9.2
+zipp                     3.17.0

requirements.txt

@@ -0,0 +1,13 @@
+torch>=2.0.0
+torchvision
+cupy-cuda12x
+transformers==4.46.2
+controlnet-aux==0.0.7
+imageio
+imageio[ffmpeg]
+safetensors
+einops
+sentencepiece
+protobuf
+modelscope
+ftfy