Update README.md

README.md

@@ -8,118 +8,202 @@ tags:
 - spatial intelligence
 - 3D vision
 - depth estimation
+- multi-view generation
+- camera calibration
 pipeline_tag: image-to-3d
 license: other
 ---
 
-# **Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States**
+<p align="center">
+  <img src="https://raw.githubusercontent.com/KangLiao929/Puffin/main/Puffin-World/assets/Puffin-World-logo_transparent.png" width="240" alt="Puffin-World logo">
+</p>
 
-## Abstract
+<h1 align="center">Puffin-World: Scaling a Unified Multimodal Model with Native 3D World States</h1>
 
-We introduce **Puffin-World**, a unified multimodal world model that represents the physical environment through three complementary **native 3D world states** — **physics** (gravity-aware camera understanding and physically consistent trajectory propagation), **geometry** (dense spatial structure for view synthesis and 3D reconstruction), and **appearance** (high-fidelity, spatially coherent visual content). A single integrated framework of vision encoder, LLM, and diffusion model — without task-specific external modules — supports physical-world perception, free-viewpoint spatial simulation, 3D world modeling, and closed-loop interaction. At its core is camera-centric **multi-view world modeling**: given one initial view and a camera trajectory, generate the remaining views jointly in RGB and depth, with 3D point-cloud reconstruction. Puffin-World is trained on **Puffin-16M**, 15M vision-language-camera triplets and 1M diverse camera trajectories curated from 28 public datasets. Built on [Puffin](https://github.com/KangLiao929/Puffin) (*Thinking with Camera*, ICLR 2026).
+<p align="center">
+  <a href="https://kangliao929.github.io/projects/puffin-world/">Project Page</a> ·
+  <a href="https://huggingface.co/blog/KangLiao/puffin-world">HF Article</a> ·
+  <a href="https://github.com/KangLiao929/Puffin">Code</a> ·
+  <a href="https://kangliao929.github.io/projects/puffin-16m/">Puffin-16M</a> ·
+  <a href="https://huggingface.co/collections/KangLiao/puffin-world">HF Collection</a> ·
+  <a href="https://kangliao929.github.io/projects/puffin-world/supplementary.html">Supplementary Results</a>
+</p>
 
-## Links
+<p align="center">
+  <img src="https://kangliao929.github.io/projects/puffin-world/assets/figure1-overview.webp" width="100%" alt="Overview of Puffin-World">
+</p>
 
-*   **Project Page**: [https://kangliao929.github.io/projects/puffin-world/](https://kangliao929.github.io/projects/puffin-world/)
-*   **GitHub Repository**: [https://github.com/KangLiao929/Puffin](https://github.com/KangLiao929/Puffin)
-*   **Hugging Face Dataset**: [https://huggingface.co/datasets/KangLiao/Puffin-16M](https://huggingface.co/datasets/KangLiao/Puffin-16M)
+## Overview
+
+Puffin-World is a unified multimodal world model that perceives, simulates, generates, and reconstructs the 3D world within one framework. Instead of representing a world only as a sequence of RGB frames, it explicitly models three complementary native 3D world states:
+
+- Physics: gravity fields and latitude maps that anchor observations and generated trajectories to the real world.
+- Geometry: dense depth that exposes the underlying 3D structure and supports direct reconstruction.
+- Appearance: high-fidelity images and sequences that remain spatially coherent as the camera moves.
+
+A single integrated framework of a geometry-aligned vision encoder, an LLM, a diffusion model, and a lightweight connector supports physical-world perception, free-viewpoint spatial simulation, 3D world modeling, and closed-loop interaction without task-specific external perception or reconstruction modules.
+
+At its core is camera-centric multi-view world modeling: given one initial view and a camera trajectory, Puffin-World jointly generates the remaining RGB and depth views and consolidates them into a gauge-aligned 3D point-cloud reconstruction.
+
+## Highlights
+
+- Native 3D world states: physics, geometry, and appearance are modeled together rather than leaving physical orientation and scene structure implicit.
+- Omni-Camera: a 9-channel dense camera condition combines a gravity-aware absolute perspective field with ray-based relative geometry, supporting camera-to-world grounding, rotation, translation, and compound motion.
+- Physics propagation: the gravity direction perceived from the reference view is propagated through the target camera trajectory to keep generated views in one coherent physical frame.
+- Unified understanding and generation: autoregressive camera understanding and diffusion-based generation share the same multimodal framework.
+- Scaled world modeling: Puffin-World is trained with Puffin-16M, comprising 15M vision-language-camera triplets and 1M diverse camera trajectories curated from 28 public datasets.
+
+## Results at a Glance
+
+| Capability | Result |
+|---|---|
+| Camera-to-world understanding | Best median errors in 12/12 comparisons and best AUC in 33/36 metrics, including ties, across Stanford2D3D, MegaDepth, TartanAir, and LaMAR |
+| Physical-world perception | 0.26° lowest roll error on LaMAR and 1.62° lowest vFoV error on Stanford2D3D |
+| Camera-controllable generation | 0.84° up-vector, 1.26° latitude, and 0.79° gravity median errors, with the lowest FID on Puffin-Cam-Bench |
+| 3D world modeling | Rank 1 in both PSNR (17.22) and LPIPS (0.318) on RealEstate10K |
+| Challenging trajectories | Lowest median roll and pitch errors on Puffin-Traj-Bench: 0.80° and 1.10° |
+
+See the [project page](https://kangliao929.github.io/projects/puffin-world/) and [supplementary results](https://kangliao929.github.io/projects/puffin-world/supplementary.html) for complete quantitative and qualitative evaluations.
 
 ## Model Details
 
-| | |
+| Field | Description |
 |---|---|
-| **Developed by** | Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy |
-| **Affiliation** | S-Lab, Nanyang Technological University |
-| **First released** | Preprint, 2026 |
-| **Model type** | Unified multimodal world model (autoregressive understanding + diffusion-based multi-view RGB-D generation with native 3D world states) |
-| **Modality** | Image → Text+Camera; Text+Camera → Image; Image+Trajectory → Multi-view RGB-D + 3D point cloud |
+| Developed by | Kang Liao, Yihang Luo, Xiao-Ming Wu, Linyi Jin, Size Wu, Chunyu Lin, Yao Zhao, Fei Wang, Wei Li, Chen Change Loy |
+| Affiliations | S-Lab, Nanyang Technological University; University of Michigan; Beijing Jiaotong University; ACE Robotics |
+| First released | Preprint, 2026 |
+| Model type | Unified multimodal world model with autoregressive understanding and diffusion-based multi-view RGB-D generation |
+| Modalities | Image → Text + Camera; Text + Camera → Image; Image + Trajectory → Multi-view RGB-D + 3D point cloud |
 
 ## Model Zoo
 
-| Checkpoint | LLM | Vision Encoder | Diffusion | Composition |
+| Checkpoint | LLM | Vision Encoder | Diffusion | Purpose |
 |---|---|---|---|---|
-| `Puffin-World-Base.pth` | Qwen2.5-7B | C-RADIOv3-H | SD3.5-medium | Full und+gen world model (Stage-II understanding backbone + Stage-IV asymmetric-attention RGB-D generation) |
-| `Puffin-World-Pro.pth` | Qwen2.5-1.5B | C-RADIOv4-H | SD3.5-large | Full und+gen world model (Stage-II understanding backbone + Stage-IV asymmetric-attention RGB-D generation) |
-| `Puffin-World-Caption.pth` | Qwen3.5-0.8B | C-RADIOv3-H | — | Understanding-only VLM expert (camera-aware captioning) |
+| `Puffin-World-Base.pth` | Qwen2.5-7B | C-RADIOv3-H | SD3.5-medium | Comprehensive understanding and RGB-D world modeling |
+| `Puffin-World-Pro.pth` | Qwen2.5-1.5B | C-RADIOv4-H | SD3.5-large | High-quality generation and RGB-D world modeling |
+| `Puffin-World-Caption.pth` | Qwen3.5-0.8B | C-RADIOv3-H | — | Understanding-only camera-aware captioning expert |
 
-The matching pipeline configs in the GitHub repository:
+Matching pipeline configurations:
 
-| Checkpoint | Config |
+| Checkpoint | Configuration |
 |---|---|
 | `Puffin-World-Base.pth` | `configs/pipelines/final_stage_4_world_all_asym_attn_qwen2_5_7b_radiov3H_sd3p5M.py` |
 | `Puffin-World-Pro.pth` | `configs/pipelines/final_stage_4_world_all_asym_attn_qwen2_5_1_5b_radiov4H_sd3p5L.py` |
 | `Puffin-World-Caption.pth` | `configs/pipelines/vlm_qwen3_5_0_8b_radiov3H_stage_2.py` |
 
-### Direct Use
+## Intended Use
+
+- Camera-to-world understanding from a single image, including roll, pitch, vertical field-of-view, distortion, and dense perspective fields.
+- Camera-controllable text-to-image generation across explicit camera intrinsics and orientations.
+- Image-conditioned or text-conditioned 3D world generation along long, extreme, or compound camera trajectories.
+- Joint multi-view RGB-D generation and direct 3D point-cloud reconstruction.
+- Closed-loop mimic world exploration and self-calibrated world exploration.
+
+## Installation
 
--   **Multi-view world modeling**: given one initial view and a camera trajectory, generate the remaining views jointly in RGB and depth, and reconstruct a gauge-aligned 3D point cloud.
--   **Physical-world perception**: gravity-aware single-image camera estimation (roll, pitch, vFoV, distortion) with perspective-field visualization.
--   **Free-viewpoint spatial simulation**: camera-controlled single-view generation and free-view world exploration (I2T → T2I → chunked multi-view I2I).
--   **Closed-loop interaction**: self-calibration — the model estimates its own camera anomaly and generates the corrective action sequence.
+The reference implementation uses Python 3.10, PyTorch 2.7.0, and CUDA 12.6.
 
-## Sample Usage
+```bash
+git clone https://github.com/KangLiao929/Puffin
+cd Puffin/Puffin-World
+
+conda create -n puffin-world python=3.10 -y
+conda activate puffin-world
+
+pip install torch==2.7.0 torchvision==0.22.0 torchaudio==2.7.0 \
+  --index-url https://download.pytorch.org/whl/cu126
+pip install -r requirements.txt
+pip install flash-attn==2.8.3 --no-build-isolation
+```
 
-Download the checkpoints and organize them in a `checkpoints` directory of the Puffin-World code base:
+Sanity check:
 
-```text
-Puffin/Puffin-World/
-├── checkpoints
-    ├── Puffin-World-Base.pth
-    ├── Puffin-World-Pro.pth
-    ├── Puffin-World-Caption.pth
+```bash
+export PYTHONPATH=./:$PYTHONPATH
+python -c "import torch, transformers, deepspeed, xtuner, flash_attn, trimesh; print('ok')"
 ```
 
+Download all released checkpoints:
+
 ```bash
-# pip install -U "huggingface_hub[cli]"
-huggingface-cli download KangLiao/Puffin-World --local-dir checkpoints --repo-type model
+pip install -U huggingface_hub
+hf download KangLiao/Puffin-World --local-dir checkpoints
 ```
 
-**Multi-view world modeling** (initial view + trajectory → RGB-D views + `reconstruction.glb`):
+## Quick Start
+
+Multi-view world modeling from an initial view and camera trajectory:
 
-```shell
+```bash
 export PYTHONPATH=./:$PYTHONPATH
 python scripts/demo/world_modeling.py \
-    --model Puffin-World \
-    --checkpoint checkpoints/Puffin-World-Pro.pth \
-    --dataset re10k --sample_index 0 --output output/demo_world
+  --model Puffin-World \
+  --checkpoint checkpoints/Puffin-World-Pro.pth \
+  --dataset re10k \
+  --sample_index 0 \
+  --output output/demo_world
 ```
 
-**Physical-world perception** (single-image camera estimation + perspective fields):
+Physical-world perception from a single image:
 
-```shell
+```bash
 python scripts/demo/physics_perception.py demo.jpg \
-    --model Puffin-World-Pro \
-    --checkpoint checkpoints/Puffin-World-Pro.pth \
-    --output output/demo_camera
+  --model Puffin-World-Pro \
+  --checkpoint checkpoints/Puffin-World-Pro.pth \
+  --output output/demo_camera
 ```
 
-**Camera-controlled spatial simulation**:
+Camera-controlled spatial simulation:
 
-```shell
+```bash
 python scripts/demo/spatial_simulation.py \
-    "A cozy living room with wooden furniture and warm sunlight." \
-    --model Puffin-World-Pro \
-    --checkpoint checkpoints/Puffin-World-Pro.pth \
-    --camera "0.05, -0.12, 1.2" --num 4 --vis_pf
+  "A cozy living room with wooden furniture and warm sunlight." \
+  --model Puffin-World-Pro \
+  --checkpoint checkpoints/Puffin-World-Pro.pth \
+  --camera "0.05, -0.12, 1.2" \
+  --num 4 \
+  --vis_pf
 ```
 
-**Free-view world exploration** and **self-calibration** (closed-loop):
+Free-view world exploration and self-calibration:
 
-```shell
+```bash
 python scripts/demo/freeview_world_exp.py \
-    --checkpoint checkpoints/Puffin-World-Pro.pth \
-    --input <image> --scene_dir <re10k-style scene dir> --chunk 3
+  --checkpoint checkpoints/Puffin-World-Pro.pth \
+  --input <image> \
+  --scene_dir <re10k-style-scene-dir> \
+  --chunk 3
 
 python scripts/demo/self_calibration.py <image> \
-    --checkpoint checkpoints/Puffin-World-Pro.pth \
-    --output output/demo_self_calib
+  --checkpoint checkpoints/Puffin-World-Pro.pth \
+  --output output/demo_self_calib
 ```
 
-For the 7B model, pass `--model Puffin-World-7B` (world modeling) or `--model Puffin-World-Base` (perception/simulation) with `--checkpoint checkpoints/Puffin-World-Base.pth`. Full evaluation entry points, flags and output layout: `documents/EVALUATION_World.md` in the GitHub repository.
+For the 7B model, use `--model Puffin-World-7B` for world modeling or `--model Puffin-World-Base` for perception and simulation.
+
+Detailed instructions:
+
+- [Training](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/documents/TRAINING.md)
+- [World-model evaluation and reconstruction](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/documents/EVALUATION_World.md)
+- [Camera-centric understanding and generation evaluation](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/documents/EVALUATION.md)
+- [Dataset construction and annotation](https://github.com/KangLiao929/Puffin/blob/main/Puffin-World/documents/DATASET_PIPELINE.md)
+
+## Training Data
+
+[Puffin-16M](https://kangliao929.github.io/projects/puffin-16m/) contains:
+
+- Puffin-Cam-15M: 15 million vision-language-camera triplets rendered from approximately 900,000 source panoramas across seven aspect ratios, diverse camera orientations, and vertical fields of view.
+- Puffin-Traj-1M: one million continuous trajectories covering look-up/look-down motion, clockwise and counterclockwise rotation, full 360° exploration, and compound motion.
+
+The downloadable dataset and held-out Puffin-Cam-15M-Bench and Puffin-Traj-1M-Bench splits are available in the [Puffin-16M repository](https://huggingface.co/datasets/KangLiao/Puffin-16M).
+
+## Limitations
+
+Puffin-World currently focuses on static scenes and models physics primarily through gravity and latitude. Dynamic environments, longer temporal horizons, richer object interactions, and broader physical states remain important directions for future work. Generated content may also inherit biases or failure modes from its training data and underlying pretrained components.
 
-### Citation
+## Citation
 
-If you find Puffin-World useful for your research or applications, please cite our papers using the following BibTeX:
+If Puffin-World, Puffin-16M, or the Puffin model series supports your research, please consider citing our work:
 
 ```bibtex
 @article{liao2026puffinworld,
@@ -130,13 +214,13 @@ If you find Puffin-World useful for your research or applications, please cite o
 }
 
 @article{liao2025puffin,
-  title={Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation},
-  author={Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change},
-  journal={arXiv preprint arXiv:2510.08673},
-  year={2025}
+  title   = {Thinking with Camera: A Unified Multimodal Model for Camera-Centric Understanding and Generation},
+  author  = {Liao, Kang and Wu, Size and Wu, Zhonghua and Jin, Linyi and Wang, Chao and Wang, Yikai and Wang, Fei and Li, Wei and Loy, Chen Change},
+  journal = {arXiv preprint arXiv:2510.08673},
+  year    = {2025}
 }
 ```
 
-### License
+## License
 
-This project is licensed under [NTU S-Lab License 1.0](https://github.com/KangLiao929/Puffin/blob/main/LICENSE).
+This project is released under the [NTU S-Lab License 1.0](https://github.com/KangLiao929/Puffin/blob/main/LICENSE).