PanoWorld logo PanoWorld: Real-World Panoramic Generation

Project Page arXiv Hugging Face Demo

Haoyuan Li1   Dizhe Zhang1✉   Yuemei Zhou1   Xiangkai Zhang1,2   Haoran Feng1,3   Xiaofan Lin1   Wenjie Jiang1   Bo Du4   Ming-Hsuan Yang5   Lu Qi1,4✉

1Insta360 Research    2Institute of Automation Chinese Academy of Sciences    3Tsinghua University
4Wuhan University    5UC Merced

![teaser](assets/panoworld_teaser.png)
> **Quick try:** demo assets in [`assets/demo/`](assets/demo/) — see [Inference](#inference) below. ## Updates - **[2026-07]** 🎉 Initial release of code. Training code is coming soon. ## Introduction In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation. Building on this insight, we propose **PanoWorld**, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through **Dense Panoramic Ray-Conditioning (DPRC)** and **Geometry-aware Memory Augmentation (GMA)**. Then, a three-stage training pipeline is introduced to progressively optimize each component. To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct **World360**, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by **AirSim360**. ## Environment Setup ### Prerequisites - **OS**: Linux (tested on Ubuntu 22.04) - **GPU**: CUDA-compatible GPU with ≥20GB VRAM - **CUDA**: 12.8 or higher - **Python**: 3.10 - **FFmpeg**: For video processing ### Step 1: Create Conda Environment Action Model inference assumes the **`PanoWorld`** conda env is already active. ```bash git clone https://github.com/Insta360-Research-Team/PanoWorld.git cd PanoWorld bash scripts/setup_panoworld_env.sh conda activate PanoWorld ``` Or install manually: ```bash conda create -n PanoWorld python=3.10 -y conda activate PanoWorld pip install -e . export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}" ``` Dependencies are listed in [`requirements.txt`](requirements.txt). After Step 1, continue with **Step 2–3** (base model + PanoWorld checkpoints) before running [`inference_action.sh`](inference_action.sh). ### Step 2: Download Base Model (Wan2.2-TI2V-5B) PanoWorld is built upon the [Wan-AI Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) video diffusion model. ```bash # Download using provided script python scripts/download_wan2.2.py # Or manually download from Hugging Face # Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B # Download to: models/Wan-AI/Wan2.2-TI2V-5B/ ``` ### Step 3: Download Panoworld Models Download the 480p or 720p checkpoints: ```bash # Option 1: Using our download script python scripts/download_panoworld_models.py # Option 2: Manual download from Hugging Face # Visit: https://huggingface.co/Insta360-Research/PanoWorld ``` Final model directory structure: ``` models/ ├── Wan-AI/ │ └── Wan2.2-TI2V-5B/ ├── lora/ │ ├── 480p/480p_lora.safetensors │ └── 720p/720p_lora.safetensors ├── action/ │ ├── 480p/480p_action.safetensors │ └── 720p/720p_action.safetensors └── casual/ ├── dmd_chunkwise/model.pt └── rolling_dmd_480p_161/model.pt ``` ### Step 4: Install Causal-Forcing Package (Optional) The Causal-Forcing stage requires additional dependencies: Please refer to [Causal-Forcing/README.md](Casual-forcing/README.md) for installation instructions. ## World360 Dataset ![teaser](assets/dataset_13x13.gif) World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information. **Output**: Each dataset generates: - 81-frame & 161-frame panoramic video. - Camera trajectory csv file - PNG image sequence ## Inference Bundled demo assets live under [`assets/demo/`](assets/demo/). Resolution-specific cases are under `480/` and `720/`; each case folder contains a **2:1 equirectangular** panorama (`input.jpg`), text prompt (`prompt.txt`), and camera trajectory (`pose.txt`). 720p cases also include a reference video (`reference_gen_joint_step2000.mp4`). > **Pose spacing:** For custom `pose.txt` trajectories, we recommend a horizontal step of **~0.05** between consecutive frames, i.e. `sqrt(action_dx² + action_dy²) ≈ 0.05` in the model world frame (`-x` forward, `+y` left). The bundled demo poses use this spacing. ### Camera-Controlled Video Generation High-quality panoramic I2V with **camera trajectory control**. Unified entry: [`inference_action.sh`](inference_action.sh) **Output:** `{output}/gen_video.mp4` (single sample) or `gen_video_{i}.mp4` (batch). ```bash # --- Demo: image + prompt + synthetic forward motion --- ./inference_action.sh \ --resolution 480 \ --image assets/demo/input.jpg \ --prompt "$(cat assets/demo/prompt.txt)" \ --motion forward \ --output ./inference_output/demo_forward # --- Demo case with recorded pose (480p) --- CASE=assets/demo/480/case2_waterway_slice000 ./inference_action.sh \ --resolution 480 \ --image ${CASE}/input.jpg \ --prompt "$(cat ${CASE}/prompt.txt)" \ --motion ${CASE}/pose.txt \ --output ${CASE}/out_action # --- Demo case with recorded pose (720p) --- CASE=assets/demo/720/case1_waterway_slice706 ./inference_action.sh \ --resolution 720 \ --image ${CASE}/input.jpg \ --prompt "$(cat ${CASE}/prompt.txt)" \ --motion ${CASE}/pose.txt \ --output ${CASE}/out_action # 480p — single sample from test CSV ./inference_action.sh \ --resolution 480 \ --prompt_nums 1 \ --output ./inference_output/demo_480p # 720p — native 1408×704, upscaled to 1440×720 ./inference_action.sh \ --resolution 720 \ --prompt_nums 1 \ --output ./inference_output/demo_720p # Or via top-level wrapper RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh python Action-Model/infer_action.py --help ``` | Flag | Description | |------|-------------| | `--resolution` | `480` (960×480) or `720` (1408×704 → 1440×720) | | `--image` / `--prompt` | Demo mode: panoramic image + text prompt (2:1 image) | | `--motion` | `forward` / `backward` / `left` / `right` / `up` /`down` /, or a pose file (demo mode) | | `--output` | Output directory | | `--output_filename` | Default `gen_video.mp4` | | `--prompt_path` | CSV with columns `video`, `short_prompt`, `pose_path` (batch mode) | | `--prompt_nums` | Number of CSV rows to run (batch mode) | Default test CSVs: `data_test.csv` (480p), `data_test_720p.csv` (720p). --- ### Causal Forcing Stage Real-time panoramic generation with **Causal Forcing** on Wan2.2 5B. Entry: [`Casual-forcing/inference_causal.sh`](Casual-forcing/inference_causal.sh) (requires `causal_forcing` env). **Output:** `{output}/causal_video.mp4` ```bash cd Casual-forcing # --- Quick demo (root assets, 480p case1) --- ./inference_causal.sh \ --image ../assets/demo/input.jpg \ --prompt "$(cat ../assets/demo/prompt.txt)" \ --output ../assets/demo/out_short # --- Demo case with recorded pose --- CASE=../assets/demo/480/case2_waterway_slice000 ./inference_causal.sh \ --image ${CASE}/input.jpg \ --prompt "$(cat ${CASE}/prompt.txt)" \ --motion ${CASE}/pose.txt \ --output ${CASE}/out_pose # --- 161-frame long video (Rolling Forcing) --- CASE=../assets/demo/480/case1_park_slice002 ./inference_causal.sh \ --image ${CASE}/input.jpg \ --prompt "$(cat ${CASE}/prompt.txt)" \ --motion ${CASE}/pose.txt \ --frames 161 \ --output ${CASE}/out_long # --- Synthetic motion presets --- ./inference_causal.sh \ --image ../assets/demo/input.jpg \ --prompt "$(cat ../assets/demo/prompt.txt)" \ --motion forward \ --output ../assets/demo/out_forward python infer_causal.py --help ``` | Flag | Description | |------|-------------| | `--frames` | `81` (short, default) or `161` (long / Rolling Forcing) | | `--motion` | `forward` / `backward` / `left` / `right`, or a pose file (`.txt` / `.csv` / `.npy`) | | `--output` | Output directory; video is always `causal_video.mp4` | --- ### Memory Preview > **Coming soon.** Long-range memory-augmented 720p inference (161-frame) will be released in a future update. --- ## Training PanoWorld training is organized in stages: ``` Stage 1 Video LoRA → output/lora_480p | lora_720p Stage 2 Action Model → output/action_480p | action_720p (requires Stage 1) Stage 3 Memory → coming soon Stage 4 Causal Forcing → Casual-forcing/logs/ (480p, see below) ``` Use `RESOLUTION=480|720` to switch presets (Matrix-3D convention). Config: [`configs/resolution.yaml`](configs/resolution.yaml). Training CSVs must include panoramic video paths and text prompts. Override defaults with `DATA_CSV=/path/to/your.csv`. | Resolution | Native train | Output | LoRA rank | |------------|-------------|--------|-----------| | **480** | 960×480 | 960×480 | 64 | | **720** | 1408×704 | 1440×720 (inference upscale) | 256 | --- ### Stage 1 — Video LoRA Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module). ```bash # 480p RESOLUTION=480 bash scripts/train/01_video_lora.sh # 720p (recommended for Action Model 720p) RESOLUTION=720 bash scripts/train/01_video_lora.sh ``` Checkpoints are saved under `{OUTPUT}/checkpoints/` with `latest.json` for resume. Legacy wrappers: `scripts/train/01_video_lora_480p.sh`, `01_video_lora_720p.sh`. --- ### Stage 2 — Action Model Freeze LoRA and train the `cam_self_attn` camera-control module (trajectory-conditioned I2V). **Requires Stage 1 LoRA.** ```bash # After Stage 1 completes RESOLUTION=480 bash scripts/train/02_action.sh RESOLUTION=720 bash scripts/train/02_action.sh # Warm-start from a simulation-pretrained action checkpoint (optional) INIT_ACTION=/path/to/simulation_action.safetensors \ RESOLUTION=720 bash scripts/train/02_action.sh ``` Checkpoints: `{OUTPUT}/checkpoints/`. Use the latest action weights with [`inference_action.sh`](inference_action.sh) for preview inference. 720p project-specific script (hardcoded paths): [`Action-Model/train_720p.sh`](Action-Model/train_720p.sh). --- ### Stage 3 — Memory > **Coming soon.** --- ### Stage 4 — Causal Forcing See [Casual-forcing/README.md](Casual-forcing/README.md) for installation and training. ### Checkpoint Layout ``` output/ ├── lora_480p/checkpoints/ ├── lora_720p/checkpoints/ ├── action_480p/checkpoints/ └── action_720p/checkpoints/ Casual-forcing/logs/ # Causal Forcing training outputs ``` ## Project Structure ``` PanoWorld/ ├── assets/demo/ # Demo inputs (480/720 cases: input, prompt, pose) │ ├── 480/ # 960×480 cases │ └── 720/ # 1408×704 cases (+ reference_gen_*.mp4) ├── inference_action.sh # Action Model unified entry → gen_video.mp4 ├── Action-Model/ # Action Model Python scripts (480p / 720p) │ └── infer_action.py # unified CLI ├── Casual-forcing/ # Causal Forcing inference & training │ ├── infer_causal.py # unified CLI → causal_video.mp4 │ └── inference_causal.sh ├── diffsynth/ # Shared DiffSynth modules ├── scripts/train/ # Staged training launchers (01–04) ├── models/ # Wan2.2 base + PanoWorld checkpoints │ ├── lora/ # Video LoRA (480p / 720p) │ ├── action/ # Action Model (480p / 720p) │ └── casual/ # Causal Forcing inference weights ├── inference_preview.sh # RESOLUTION=480|720 wrapper ├── utils/ │ ├── run_metrics.sh # Unified 480p/720p quality evaluation │ ├── resolution_config.py # Resolution presets helper │ └── validate/ # step1_validate, step2_tractory, metric modules └── configs/resolution.yaml # 480 / 720 presets ``` ## Tools **Panoramic video quality evaluation** — unified script for 480p / 720p: [`utils/run_metrics.sh`](utils/run_metrics.sh) (implementation: [`utils/validate/`](utils/validate/)). Metrics: **PSNR** (windowed), **FID** / **FID_pole** / **FID_equ**, **FAED**, **NIQE**, **QAlign** (Quality / Aesthetic). ```bash RESOLUTION=480 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh RESOLUTION=720 INFERENCE_DIR=/path/to/inference RESULT_DIR=/path/to/result bash utils/run_metrics.sh ``` Results are written to `RESULT_DIR` (PSNR CSV + `eval_results/eval_results_*.txt`). ## Acknowledgments We thank the following projects for their inspiring work, our code is partially based on the code from these projects: - **[Wan-AI](https://huggingface.co/Wan-AI)**: Base video diffusion model - **[UCPE](https://github.com/chengzhag/UCPE)**: Camera-controlled text-to-video generation. - **[Causal-Forcing](https://github.com/thu-ml/Causal-Forcing)**: Causal-Forcing distillation for fast diffusion models ## Citation If you find Panoworld useful for your research, please cite: ```bibtex @misc{li2026panoworldrealworldpanoramicgeneration, title={PanoWorld: Real-World Panoramic Generation}, author={Haoyuan Li and Dizhe Zhang and Yuemei Zhou and Xiangkai Zhang and Haoran Feng and Xiaofan Lin and Wenjie Jiang and Bo Du and Ming-Hsuan Yang and Lu Qi}, year={2026}, eprint={2607.09661}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2607.09661}, } ```