|
|
|
|
|
|
| <div align="center" style="margin-top: 0px; margin-bottom: 0px;"> |
| <h1> |
| <img src="assets/helicopter.png" width="64" alt="PanoWorld logo" style="vertical-align: middle; margin-right: 12px;"> |
| PanoWorld: Real-World Panoramic Generation |
| </h1> |
|
|
| <p align="center"> |
| <a href="https://lihaoy-ux.github.io/panoworld-page/"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a> |
| <a href="http://arxiv.org/abs/2607.09661"><img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv" alt="arXiv"></a> |
| <a href="https://huggingface.co/Insta360-Research/PanoWorld/"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="Hugging Face"></a> |
| <a href="#"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-World360(coming soon)-yellow" alt="Demo"></a> |
| </p> |
|
|
| <p align="center"> |
| Haoyuan Li<sup>1</sup> Dizhe Zhang<sup>1</sup>✉ Yuemei Zhou<sup>1</sup> Xiangkai Zhang<sup>1,2</sup> Haoran Feng<sup>1,3</sup> Xiaofan Lin<sup>1</sup> Wenjie Jiang<sup>1</sup> Bo Du<sup>4</sup> Ming-Hsuan Yang<sup>5</sup> Lu Qi<sup>1,4</sup>✉ |
| <br><br> |
| <sup>1</sup>Insta360 Research |
| <sup>2</sup>Institute of Automation Chinese Academy of Sciences |
| <sup>3</sup>Tsinghua University |
| <br> |
| <sup>4</sup>Wuhan University |
| <sup>5</sup>UC Merced |
| </p> |
|
|
|  |
|
|
|
|
|
|
| </div> |
|
|
| > **Quick try:** demo assets in [`assets/demo/`](assets/demo/) — see [Inference](#inference) below. |
|
|
|
|
| ## Updates |
|
|
| - **[2026-07]** 🎉 Initial release of code. Training code is coming soon. |
|
|
| ## Introduction |
|
|
| In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation. |
|
|
| Building on this insight, we propose **PanoWorld**, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through **Dense Panoramic Ray-Conditioning (DPRC)** and **Geometry-aware Memory Augmentation (GMA)**. Then, a three-stage training pipeline is introduced to progressively optimize each component. |
|
|
| To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct **World360**, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by **AirSim360**. |
|
|
| ## Environment Setup |
|
|
| ### Prerequisites |
|
|
| - **OS**: Linux (tested on Ubuntu 22.04) |
| - **GPU**: CUDA-compatible GPU with ≥20GB VRAM |
| - **CUDA**: 12.8 or higher |
| - **Python**: 3.10 |
| - **FFmpeg**: For video processing |
|
|
| ### Step 1: Create Conda Environment |
|
|
| Action Model inference assumes the **`PanoWorld`** conda env is already active. |
|
|
| ```bash |
| git clone https://github.com/Insta360-Research-Team/PanoWorld.git |
| cd PanoWorld |
| |
| bash scripts/setup_panoworld_env.sh |
| conda activate PanoWorld |
| ``` |
|
|
| Or install manually: |
|
|
| ```bash |
| conda create -n PanoWorld python=3.10 -y |
| conda activate PanoWorld |
| pip install -e . |
| export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}" |
| ``` |
|
|
| Dependencies are listed in [`requirements.txt`](requirements.txt). After Step 1, continue with **Step 2–3** (base model + PanoWorld checkpoints) before running [`inference_action.sh`](inference_action.sh). |
|
|
| ### Step 2: Download Base Model (Wan2.2-TI2V-5B) |
|
|
| PanoWorld is built upon the [Wan-AI Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) video diffusion model. |
|
|
| ```bash |
| # Download using provided script |
| python scripts/download_wan2.2.py |
| |
| # Or manually download from Hugging Face |
| # Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B |
| # Download to: models/Wan-AI/Wan2.2-TI2V-5B/ |
| ``` |
|
|
| ### Step 3: Download Panoworld Models |
|
|
| Download the 480p or 720p checkpoints: |
|
|
| ```bash |
| # Option 1: Using our download script |
| python scripts/download_panoworld_models.py |
| |
| # Option 2: Manual download from Hugging Face |
| # Visit: https://huggingface.co/Insta360-Research/PanoWorld |
| ``` |
|
|
| Final model directory structure: |
|
|
| ``` |
| models/ |
| ├── Wan-AI/ |
| │ └── Wan2.2-TI2V-5B/ |
| ├── lora/ |
| │ ├── 480p/480p_lora.safetensors |
| │ └── 720p/720p_lora.safetensors |
| ├── action/ |
| │ ├── 480p/480p_action.safetensors |
| │ └── 720p/720p_action.safetensors |
| └── casual/ |
| ├── dmd_chunkwise/model.pt |
| └── rolling_dmd_480p_161/model.pt |
| ``` |
|
|
| ### Step 4: Install Causal-Forcing Package (Optional) |
|
|
| The Causal-Forcing stage requires additional dependencies: |
|
|
| Please refer to [Causal-Forcing/README.md](Casual-forcing/README.md) for installation instructions. |
|
|
|
|
| ## World360 Dataset |
|
|
|
|
|  |
| World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information. |
|
|
|
|
|
|
|
|
| **Output**: Each dataset generates: |
|
|
| - 81-frame & 161-frame panoramic video. |
| - Camera trajectory csv file |
| - PNG image sequence |
|
|
|
|
|
|
|
|
| ## Inference |
|
|
| Bundled demo assets live under [`assets/demo/`](assets/demo/). Resolution-specific cases are under `480/` and `720/`; each case folder contains a **2:1 equirectangular** panorama (`input.jpg`), text prompt (`prompt.txt`), and camera trajectory (`pose.txt`). 720p cases also include a reference video (`reference_gen_joint_step2000.mp4`). |
|
|
| > **Pose spacing:** For custom `pose.txt` trajectories, we recommend a horizontal step of **~0.05** between consecutive frames, i.e. `sqrt(action_dx² + action_dy²) ≈ 0.05` in the model world frame (`-x` forward, `+y` left). The bundled demo poses use this spacing. |
|
|
|
|
|
|
|
|
| ### Camera-Controlled Video Generation |
|
|
| High-quality panoramic I2V with **camera trajectory control**. |
| Unified entry: [`inference_action.sh`](inference_action.sh) |
|
|
| **Output:** `{output}/gen_video.mp4` (single sample) or `gen_video_{i}.mp4` (batch). |
|
|
| ```bash |
| # --- Demo: image + prompt + synthetic forward motion --- |
| ./inference_action.sh \ |
| --resolution 480 \ |
| --image assets/demo/input.jpg \ |
| --prompt "$(cat assets/demo/prompt.txt)" \ |
| --motion forward \ |
| --output ./inference_output/demo_forward |
| |
| # --- Demo case with recorded pose (480p) --- |
| CASE=assets/demo/480/case2_waterway_slice000 |
| ./inference_action.sh \ |
| --resolution 480 \ |
| --image ${CASE}/input.jpg \ |
| --prompt "$(cat ${CASE}/prompt.txt)" \ |
| --motion ${CASE}/pose.txt \ |
| --output ${CASE}/out_action |
| |
| # --- Demo case with recorded pose (720p) --- |
| CASE=assets/demo/720/case1_waterway_slice706 |
| ./inference_action.sh \ |
| --resolution 720 \ |
| --image ${CASE}/input.jpg \ |
| --prompt "$(cat ${CASE}/prompt.txt)" \ |
| --motion ${CASE}/pose.txt \ |
| --output ${CASE}/out_action |
| |
| # 480p — single sample from test CSV |
| ./inference_action.sh \ |
| --resolution 480 \ |
| --prompt_nums 1 \ |
| --output ./inference_output/demo_480p |
| |
| # 720p — native 1408×704, upscaled to 1440×720 |
| ./inference_action.sh \ |
| --resolution 720 \ |
| --prompt_nums 1 \ |
| --output ./inference_output/demo_720p |
| |
| # Or via top-level wrapper |
| RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh |
| RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh |
| |
| python Action-Model/infer_action.py --help |
| ``` |
|
|
| | Flag | Description | |
| |------|-------------| |
| | `--resolution` | `480` (960×480) or `720` (1408×704 → 1440×720) | |
| | `--image` / `--prompt` | Demo mode: panoramic image + text prompt (2:1 image) | |
| | `--motion` | `forward` / `backward` / `left` / `right` / `up` /`down` /, or a pose file (demo mode) | |
| | `--output` | Output directory | |
| | `--output_filename` | Default `gen_video.mp4` | |
| | `--prompt_path` | CSV with columns `video`, `short_prompt`, `pose_path` (batch mode) | |
| | `--prompt_nums` | Number of CSV rows to run (batch mode) | |
|
|
| Default test CSVs: `data_test.csv` (480p), `data_test_720p.csv` (720p). |
|
|
| --- |
|
|
| ### Causal Forcing Stage |
| Real-time panoramic generation with **Causal Forcing** on Wan2.2 5B. |
| Entry: [`Casual-forcing/inference_causal.sh`](Casual-forcing/inference_causal.sh) (requires `causal_forcing` env). |
|
|
| **Output:** `{output}/causal_video.mp4` |
|
|
| ```bash |
| cd Casual-forcing |
| |
| # --- Quick demo (root assets, 480p case1) --- |
| ./inference_causal.sh \ |
| --image ../assets/demo/input.jpg \ |
| --prompt "$(cat ../assets/demo/prompt.txt)" \ |
| --output ../assets/demo/out_short |
| |
| # --- Demo case with recorded pose --- |
| CASE=../assets/demo/480/case2_waterway_slice000 |
| ./inference_causal.sh \ |
| --image ${CASE}/input.jpg \ |
| --prompt "$(cat ${CASE}/prompt.txt)" \ |
| --motion ${CASE}/pose.txt \ |
| --output ${CASE}/out_pose |
| |
| # --- 161-frame long video (Rolling Forcing) --- |
| CASE=../assets/demo/480/case1_park_slice002 |
| ./inference_causal.sh \ |
| --image ${CASE}/input.jpg \ |
| --prompt "$(cat ${CASE}/prompt.txt)" \ |
| --motion ${CASE}/pose.txt \ |
| --frames 161 \ |
| --output ${CASE}/out_long |
| |
| # --- Synthetic motion presets --- |
| ./inference_causal.sh \ |
| --image ../assets/demo/input.jpg \ |
| --prompt "$(cat ../assets/demo/prompt.txt)" \ |
| --motion forward \ |
| --output ../assets/demo/out_forward |
| |
| python infer_causal.py --help |
| ``` |
|
|
| | Flag | Description | |
| |------|-------------| |
| | `--frames` | `81` (short, default) or `161` (long / Rolling Forcing) | |
| | `--motion` | `forward` / `backward` / `left` / `right`, or a pose file (`.txt` / `.csv` / `.npy`) | |
| | `--output` | Output directory; video is always `causal_video.mp4` | |
| --- |
| |
| ### Memory Preview |
| |
| > **Coming soon.** Long-range memory-augmented 720p inference (161-frame) will be released in a future update. |
| |
| --- |
| |
| ## Training |
| |
| PanoWorld training is organized in stages: |
| |
| ``` |
| Stage 1 Video LoRA → output/lora_480p | lora_720p |
| Stage 2 Action Model → output/action_480p | action_720p (requires Stage 1) |
| Stage 3 Memory → coming soon |
| Stage 4 Causal Forcing → Casual-forcing/logs/ (480p, see below) |
| ``` |
| |
| Use `RESOLUTION=480|720` to switch presets (Matrix-3D convention). Config: [`configs/resolution.yaml`](configs/resolution.yaml). |
| |
| |
| |
| Training CSVs must include panoramic video paths and text prompts. Override defaults with `DATA_CSV=/path/to/your.csv`. |
| |
| | Resolution | Native train | Output | LoRA rank | |
| |------------|-------------|--------|-----------| |
| | **480** | 960×480 | 960×480 | 64 | |
| | **720** | 1408×704 | 1440×720 (inference upscale) | 256 | |
| |
| --- |
|
|
| ### Stage 1 — Video LoRA |
|
|
| Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module). |
|
|
| ```bash |
| # 480p |
| RESOLUTION=480 bash scripts/train/01_video_lora.sh |
| |
| # 720p (recommended for Action Model 720p) |
| RESOLUTION=720 bash scripts/train/01_video_lora.sh |
| ``` |
|
|
|
|
| Checkpoints are saved under `{OUTPUT}/checkpoints/` with `latest.json` for resume. |
|
|
| Legacy wrappers: `scripts/train/01_video_lora_480p.sh`, `01_video_lora_720p.sh`. |
|
|
| --- |
|
|
| ### Stage 2 — Action Model |
|
|
| Freeze LoRA and train the `cam_self_attn` camera-control module (trajectory-conditioned I2V). **Requires Stage 1 LoRA.** |
|
|
| ```bash |
| # After Stage 1 completes |
| RESOLUTION=480 bash scripts/train/02_action.sh |
| RESOLUTION=720 bash scripts/train/02_action.sh |
| |
| # Warm-start from a simulation-pretrained action checkpoint (optional) |
| INIT_ACTION=/path/to/simulation_action.safetensors \ |
| RESOLUTION=720 bash scripts/train/02_action.sh |
| ``` |
|
|
|
|
| Checkpoints: `{OUTPUT}/checkpoints/`. Use the latest action weights with [`inference_action.sh`](inference_action.sh) for preview inference. |
|
|
| 720p project-specific script (hardcoded paths): [`Action-Model/train_720p.sh`](Action-Model/train_720p.sh). |
|
|
| --- |
|
|
| ### Stage 3 — Memory |
|
|
| > **Coming soon.** |
|
|
| --- |
|
|
| ### Stage 4 — Causal Forcing |
|
|
| See [Casual-forcing/README.md](Casual-forcing/README.md) for installation and training. |
|
|
| ### Checkpoint Layout |
|
|
| ``` |
| output/ |
| ├── lora_480p/checkpoints/ |
| ├── lora_720p/checkpoints/ |
| ├── action_480p/checkpoints/ |
| └── action_720p/checkpoints/ |
| Casual-forcing/logs/ # Causal Forcing training outputs |
| ``` |
|
|
| ## Project Structure |
|
|
| ``` |
| PanoWorld/ |
| ├── assets/demo/ # Demo inputs (480/720 cases: input, prompt, pose) |
| │ ├── 480/ # 960×480 cases |
| │ └── 720/ # 1408×704 cases (+ reference_gen_*.mp4) |
| ├── inference_action.sh # Action Model unified entry → gen_video.mp4 |
| ├── Action-Model/ # Action Model Python scripts (480p / 720p) |
| │ └── infer_action.py # unified CLI |
| ├── Casual-forcing/ # Causal Forcing inference & training |
| │ ├── infer_causal.py # unified CLI → causal_video.mp4 |
| │ └── inference_causal.sh |
| ├── diffsynth/ # Shared DiffSynth modules |
| ├── scripts/train/ # Staged training launchers (01–04) |
| ├── models/ # Wan2.2 base + PanoWorld checkpoints |
| │ ├── lora/ # Video LoRA (480p / 720p) |
| │ ├── action/ # Action Model (480p / 720p) |
| │ └── casual/ # Causal Forcing inference weights |
| ├── inference_preview.sh # RESOLUTION=480|720 wrapper |
| ├── utils/ |
| │ ├── run_metrics.sh # Unified 480p/720p quality evaluation |
| │ ├── resolution_config.py # Resolution presets helper |
| │ └── validate/ # step1_validate, step2_tractory, metric modules |
| └── configs/resolution.yaml # 480 / 720 presets |
| ``` |
|
|
| ## Tools |
|
|
| **Panoramic video quality evaluation** — unified script for 480p / 720p: [`utils/run_metrics.sh`](utils/run_metrics.sh) (implementation: [`utils/validate/`](utils/validate/)). |
|
|
| Metrics: **PSNR** (windowed), **FID** / **FID_pole** / **FID_equ**, **FAED**, **NIQE**, **QAlign** (Quality / Aesthetic). |
|
|
| ```bash |
| RESOLUTION=480 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh |
| RESOLUTION=720 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh |
| ``` |
|
|
| Results are written to `RESULT_DIR` (PSNR CSV + `eval_results/eval_results_*.txt`). |
|
|
| ## Acknowledgments |
|
|
| We thank the following projects for their inspiring work, our code is partially based on the code from these projects: |
| - **[Wan-AI](https://huggingface.co/Wan-AI)**: Base video diffusion model |
|
|
| - **[UCPE](https://github.com/chengzhag/UCPE)**: Camera-controlled text-to-video generation. |
| - **[Causal-Forcing](https://github.com/thu-ml/Causal-Forcing)**: Causal-Forcing distillation for fast diffusion models |
|
|
|
|
| ## Citation |
|
|
| If you find Panoworld useful for your research, please cite: |
|
|
| ```bibtex |
| @misc{li2026panoworldrealworldpanoramicgeneration, |
| title={PanoWorld: Real-World Panoramic Generation}, |
| author={Haoyuan Li and Dizhe Zhang and Yuemei Zhou and Xiangkai Zhang and Haoran Feng and Xiaofan Lin and Wenjie Jiang and Bo Du and Ming-Hsuan Yang and Lu Qi}, |
| year={2026}, |
| eprint={2607.09661}, |
| archivePrefix={arXiv}, |
| primaryClass={cs.CV}, |
| url={https://arxiv.org/abs/2607.09661}, |
| } |
| ``` |
|
|
|
|
|
|
|
|
