Update README.md

README.md

@@ -0,0 +1,397 @@
+
+
+
+<div align="center" style="margin-top: 0px; margin-bottom: 0px;">
+<h1>
+  <img src="assets/helicopter.png" width="64" alt="PanoWorld logo" style="vertical-align: middle; margin-right: 12px;">
+  PanoWorld: Real-World Panoramic Generation
+</h1>
+
+<p align="center">
+  <a href="https://lihaoy-ux.github.io/panoworld-page/"><img src="https://img.shields.io/badge/Project-Page-green" alt="Project Page"></a>
+  <a href="https://arxiv.org/pdf/2607.08765v1"><img src="https://img.shields.io/badge/arXiv-Paper-red?logo=arxiv" alt="arXiv"></a>
+  <a href="#"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model (coming soon)-blue" alt="Hugging Face"></a>
+  <a href="#"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Demo (coming soon)-yellow" alt="Demo"></a>
+</p>
+
+![teaser](assets/panoworld_teaser.png)
+
+
+
+</div>
+
+OmniRoam Teaser
+
+> **Quick try:** demo assets in [`assets/demo/`](assets/demo/) — see [Inference](#inference) below.
+
+
+## Updates
+
+- **[2026-07]** 🎉 Initial release of code
+
+## Introduction
+
+In this work, we aim to address the challenge of long-range memory in panoramic world models by exploiting the rotation-equivariant property of omnidirectional representations, where rotation can be treated as an implicit geometric transformation.
+
+Building on this insight, we propose **PanoWorld**, which simplifies camera trajectories into translations via fixed headings for both current-action modeling and long-range memory through **Dense Panoramic Ray-Conditioning (DPRC)** and **Geometry-aware Memory Augmentation (GMA)**. Then, a three-stage training pipeline is introduced to progressively optimize each component.
+
+To better evaluate physical consistency under large-scale spatial variations and diverse illumination conditions, where existing datasets are relatively stable, we construct **World360**, a large-scale dataset consisting of both real-world video clips collected via panoramic unmanned aerial vehicles and high-quality simulated clips generated by **AirSim360**.
+
+## Environment Setup
+
+### Prerequisites
+
+- **OS**: Linux (tested on Ubuntu 22.04)
+- **GPU**: CUDA-compatible GPU with ≥20GB VRAM
+- **CUDA**: 12.8 or higher
+- **Python**: 3.10
+- **FFmpeg**: For video processing
+
+### Step 1: Create Conda Environment
+
+Action Model inference uses the **`PanoWorld`** conda env (see [`inference_action.sh`](inference_action.sh)).
+
+```bash
+git clone https://github.com/Insta360-Research-Team/PanoWorld.git
+cd PanoWorld
+
+bash scripts/setup_panoworld_env.sh
+conda activate PanoWorld
+```
+
+Or install manually:
+
+```bash
+conda create -n PanoWorld python=3.10 -y
+conda activate PanoWorld
+pip install -e .
+export PYTHONPATH="$(pwd)${PYTHONPATH:+:${PYTHONPATH}}"
+```
+
+Dependencies are listed in [`requirements.txt`](requirements.txt). After Step 1, continue with **Step 2–3** (base model + PanoWorld checkpoints) before running [`inference_action.sh`](inference_action.sh).
+
+### Step 2: Download Base Model (Wan2.2-TI2V-5B)
+
+PanoWorld is built upon the [Wan-AI Wan2.2-TI2V-5B](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B) video diffusion model.
+
+```bash
+# Download using provided script
+python scripts/download_wan2.2.py
+
+# Or manually download from Hugging Face
+# Visit: https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B
+# Download to: models/Wan-AI/Wan2.2-TI2V-5B/
+```
+
+### Step 3: Download Panoworld Models
+
+Download the 480p or 720p checkpoints:
+
+```bash
+# Option 1: Using our download script
+python scripts/download_panoworld_models.py
+
+# Option 2: Manual download from Hugging Face
+# Visit: https://huggingface.co/Insta360-Research/PanoWorld
+```
+
+Final model directory structure:
+
+```
+models/
+├── Wan-AI/
+│   └── Wan2.2-TI2V-5B/
+├── lora/
+│   ├── 480p/480p_lora.safetensors
+│   └── 720p/720p_lora.safetensors
+├── action/
+│   ├── 480p/480p_action.safetensors
+│   └── 720p/720p_action.safetensors
+└── casual/
+    ├── dmd_chunkwise/model.pt
+    └── rolling_dmd_480p_161/model.pt
+```
+
+### Step 4: Install Causal-Forcing Package (Optional)
+
+The Causal-Forcing stage requires additional dependencies:
+
+Please refer to [Causal-Forcing/README.md](Casual-forcing/README.md) for installation instructions.
+
+
+## World360 Dataset
+
+
+![teaser](assets/dataset_13x13.gif)
+World360 comprises 120,000 high-quality sequences, unifying 70,000 curated real-world clips with 50,000 high-fidelity simulations from AirSim360, and introduces diverse multi-altitude aerial trajectories with precise camera poses and depth information. 
+
+
+
+
+**Output**: Each dataset generates:
+
+- 81-frame & 161-frame panoramic video.
+- Camera trajectory csv file
+- PNG image sequence
+
+
+
+
+## Inference
+
+Bundled demo assets live under [`assets/demo/`](assets/demo/). Resolution-specific cases are under `480/` and `720/`; each case folder contains a **2:1 equirectangular** panorama (`input.jpg`), text prompt (`prompt.txt`), and camera trajectory (`pose.txt`). 720p cases also include a reference video (`reference_gen_joint_step2000.mp4`).
+
+
+
+
+### Camera-Controlled Video Generation
+
+High-quality panoramic I2V with **camera trajectory control**.  
+Unified entry: [`inference_action.sh`](inference_action.sh) 
+
+**Output:** `{output}/gen_video.mp4` (single sample) or `gen_video_{i}.mp4` (batch).
+
+```bash
+# --- Demo: image + prompt + synthetic forward motion ---
+./inference_action.sh \
+  --resolution 480 \
+  --image assets/demo/input.jpg \
+  --prompt "$(cat assets/demo/prompt.txt)" \
+  --motion forward \
+  --output ./inference_output/demo_forward
+
+# --- Demo case with recorded pose (480p) ---
+CASE=assets/demo/480/case2_waterway_slice000
+./inference_action.sh \
+  --resolution 480 \
+  --image ${CASE}/input.jpg \
+  --prompt "$(cat ${CASE}/prompt.txt)" \
+  --motion ${CASE}/pose.txt \
+  --output ${CASE}/out_action
+
+# --- Demo case with recorded pose (720p) ---
+CASE=assets/demo/720/case1_waterway_slice706
+./inference_action.sh \
+  --resolution 720 \
+  --image ${CASE}/input.jpg \
+  --prompt "$(cat ${CASE}/prompt.txt)" \
+  --motion ${CASE}/pose.txt \
+  --output ${CASE}/out_action
+
+# 480p — single sample from test CSV
+./inference_action.sh \
+  --resolution 480 \
+  --prompt_nums 1 \
+  --output ./inference_output/demo_480p
+
+# 720p — native 1408×704, upscaled to 1440×720
+./inference_action.sh \
+  --resolution 720 \
+  --prompt_nums 1 \
+  --output ./inference_output/demo_720p
+
+# Or via top-level wrapper
+RESOLUTION=480 PROMPT_NUMS=1 ./inference_preview.sh
+RESOLUTION=720 PROMPT_NUMS=1 ./inference_preview.sh
+
+python Action-Model/infer_action.py --help
+```
+
+| Flag | Description |
+|------|-------------|
+| `--resolution` | `480` (960×480) or `720` (1408×704 → 1440×720) |
+| `--image` / `--prompt` | Demo mode: panoramic image + text prompt (2:1 image) |
+| `--motion` | `forward` / `backward` / `left` / `right` / `up` /`down` /, or a pose file (demo mode) |
+| `--output` | Output directory |
+| `--output_filename` | Default `gen_video.mp4` |
+| `--prompt_path` | CSV with columns `video`, `short_prompt`, `pose_path` (batch mode) |
+| `--prompt_nums` | Number of CSV rows to run (batch mode) |
+
+Default test CSVs: `data_test.csv` (480p), `data_test_720p.csv` (720p).
+
+---
+
+### Causal Forcing Stage
+Real-time panoramic generation with **Causal Forcing** on Wan2.2 5B.  
+Entry: [`Casual-forcing/inference_causal.sh`](Casual-forcing/inference_causal.sh) (conda env: `causal_forcing`).
+
+**Output:** `{output}/causal_video.mp4`
+
+```bash
+conda activate causal_forcing
+cd Casual-forcing
+
+# --- Quick demo (root assets, 480p case1) ---
+./inference_causal.sh \
+  --image ../assets/demo/input.jpg \
+  --prompt "$(cat ../assets/demo/prompt.txt)" \
+  --output ../assets/demo/out_short
+
+# --- Demo case with recorded pose ---
+CASE=../assets/demo/480/case2_waterway_slice000
+./inference_causal.sh \
+  --image ${CASE}/input.jpg \
+  --prompt "$(cat ${CASE}/prompt.txt)" \
+  --motion ${CASE}/pose.txt \
+  --output ${CASE}/out_pose
+
+# --- 161-frame long video (Rolling Forcing) ---
+CASE=../assets/demo/480/case1_park_slice002
+./inference_causal.sh \
+  --image ${CASE}/input.jpg \
+  --prompt "$(cat ${CASE}/prompt.txt)" \
+  --motion ${CASE}/pose.txt \
+  --frames 161 \
+  --output ${CASE}/out_long
+
+# --- Synthetic motion presets ---
+./inference_causal.sh \
+  --image ../assets/demo/input.jpg \
+  --prompt "$(cat ../assets/demo/prompt.txt)" \
+  --motion forward \
+  --output ../assets/demo/out_forward
+
+python infer_causal.py --help
+```
+
+| Flag | Description |
+|------|-------------|
+| `--frames` | `81` (short, default) or `161` (long / Rolling Forcing) |
+| `--motion` | `forward` / `backward` / `left` / `right`, or a pose file (`.txt` / `.csv` / `.npy`) |
+| `--output` | Output directory; video is always `causal_video.mp4` |
+---
+
+### Memory Preview
+
+> **Coming soon.** Long-range memory-augmented 720p inference (161-frame) will be released in a future update.
+
+---
+
+## Training
+
+PanoWorld training is organized in stages:
+
+```
+Stage 1  Video LoRA     →  output/lora_480p | lora_720p
+Stage 2  Action Model   →  output/action_480p | action_720p  (requires Stage 1)
+Stage 3  Memory         →  coming soon
+Stage 4  Causal Forcing →  Casual-forcing/logs/  (480p, see below)
+```
+
+Use `RESOLUTION=480|720` to switch presets (Matrix-3D convention). Config: [`configs/resolution.yaml`](configs/resolution.yaml).
+
+
+
+Training CSVs must include panoramic video paths and text prompts. Override defaults with `DATA_CSV=/path/to/your.csv`.
+
+| Resolution | Native train | Output | LoRA rank |
+|------------|-------------|--------|-----------|
+| **480** | 960×480 | 960×480 | 64 |
+| **720** | 1408×704 | 1440×720 (inference upscale) | 256 |
+
+---
+
+### Stage 1 — Video LoRA
+
+Fine-tune Wan2.2 I2V with LoRA for panoramic video generation (81 frames, no camera control module).
+
+```bash
+# 480p
+RESOLUTION=480 bash scripts/train/01_video_lora.sh
+
+# 720p (recommended for Action Model 720p)
+RESOLUTION=720 bash scripts/train/01_video_lora.sh
+```
+
+
+Checkpoints are saved under `{OUTPUT}/checkpoints/` with `latest.json` for resume.
+
+Legacy wrappers: `scripts/train/01_video_lora_480p.sh`, `01_video_lora_720p.sh`.
+
+---
+
+### Stage 2 — Action Model
+
+Freeze LoRA and train the `cam_self_attn` camera-control module (trajectory-conditioned I2V). **Requires Stage 1 LoRA.**
+
+```bash
+# After Stage 1 completes
+RESOLUTION=480 bash scripts/train/02_action.sh
+RESOLUTION=720 bash scripts/train/02_action.sh
+
+# Warm-start from a simulation-pretrained action checkpoint (optional)
+INIT_ACTION=/path/to/simulation_action.safetensors \
+  RESOLUTION=720 bash scripts/train/02_action.sh
+```
+
+
+Checkpoints: `{OUTPUT}/checkpoints/`. Use the latest action weights with [`inference_action.sh`](inference_action.sh) for preview inference.
+
+720p project-specific script (hardcoded paths): [`Action-Model/train_720p.sh`](Action-Model/train_720p.sh).
+
+---
+
+### Stage 3 — Memory
+
+> **Coming soon.**
+
+---
+
+### Stage 4 — Causal Forcing
+
+See [Casual-forcing/README.md](Casual-forcing/README.md) for installation and training.
+
+### Checkpoint Layout
+
+```
+output/
+├── lora_480p/checkpoints/
+├── lora_720p/checkpoints/
+├── action_480p/checkpoints/
+└── action_720p/checkpoints/
+Casual-forcing/logs/          # Causal Forcing training outputs
+```
+
+## Project Structure
+
+```
+PanoWorld/
+├── assets/demo/                # Demo inputs (480/720 cases: input, prompt, pose)
+│   ├── 480/                    # 960×480 cases
+│   └── 720/                    # 1408×704 cases (+ reference_gen_*.mp4)
+├── inference_action.sh         # Action Model unified entry → gen_video.mp4
+├── Action-Model/               # Action Model Python scripts (480p / 720p)
+│   └── infer_action.py         # unified CLI
+├── Casual-forcing/             # Causal Forcing inference & training
+│   ├── infer_causal.py         # unified CLI → causal_video.mp4
+│   └── inference_causal.sh
+├── diffsynth/                  # Shared DiffSynth modules
+├── scripts/train/              # Staged training launchers (01–04)
+├── models/                     # Wan2.2 base + PanoWorld checkpoints
+│   ├── lora/                   # Video LoRA (480p / 720p)
+│   ├── action/                 # Action Model (480p / 720p)
+│   └── casual/                 # Causal Forcing inference weights
+├── inference_preview.sh        # RESOLUTION=480|720 wrapper
+└── configs/resolution.yaml     # 480 / 720 presets
+```
+
+## Acknowledgments
+
+We thank the following projects for their inspiring work, our code is partially based on the code from these projects:
+- **[Wan-AI](https://huggingface.co/Wan-AI)**: Base video diffusion model
+
+- **[UCPE](https://github.com/chengzhag/UCPE)**: Camera-controlled text-to-video generation.
+- **[Causal-Forcing](https://github.com/thu-ml/Causal-Forcing)**: Causal-Forcing distillation for fast diffusion models
+
+
+## Citation
+
+If you find Panoworld useful for your research, please cite:
+
+```bibtex
+@article{
+}
+```
+
+
+