Upload 3 files

LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Insta360 Research Team
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

README.md

@@ -0,0 +1,287 @@
+<h1 align="center">
+UniSHARP:<br>
+Universal Sharp Monocular View Synthesis
+</h1>
+
+<p align="center">
+  <b>Meixi Song</b><sup>1</sup> ·
+  <b>Dizhe Zhang</b><sup>1,*</sup> ·
+  <b>Hao Ren</b><sup>1</sup> ·
+  <b>Ruiyang Zhang</b><sup>1</sup> ·
+  <b>Bo Du</b><sup>2</sup> ·
+  <b>Ming-Hsuan Yang</b><sup>3</sup> ·
+  <b>Lu Qi</b><sup>1,2,*</sup>
+  <br>
+  <sup>1</sup>Insta360 Research · <sup>2</sup>Wuhan University · <sup>3</sup>University of California, Merced
+</p>
+
+<p align="center">
+  <img src="https://img.shields.io/badge/arXiv-coming_soon-red?logo=arxiv&logoColor=white" alt="arXiv">
+  <a href="https://insta360-research-team.github.io/Unisharp-website/"><img src="https://img.shields.io/badge/Project_Page-Website-green" alt="Project Page"></a>
+  <a href="https://huggingface.co/spaces/Insta360-Research/UniSHARP"><img src="https://img.shields.io/badge/HuggingFace-Demo-yellow" alt="Demo"></a>
+  <a href="https://huggingface.co/datasets/Insta360-Research/OmniRooms"><img src="https://img.shields.io/badge/HuggingFace-OmniRooms-orange" alt="Dataset"></a>
+</p>
+
+UniSHARP extends SHARP-style photorealistic monocular view synthesis to universal camera systems. Given a single image from a perspective, wide-FoV, fisheye, or panoramic camera, UniSHARP predicts a 3D Gaussian representation and renders high-quality novel views.
+
+<p align="center">
+  <img src="assets/teaser.gif" alt="UniSHARP teaser" width="90%">
+</p>
+
+<p align="center">
+  <img src="assets/unisharp.png" alt="UniSHARP method" width="90%">
+</p>
+
+## Installation
+
+Clone this repository and enter the project directory:
+
+```bash
+git clone https://github.com/Insta360-Research-Team/UniSHARP.git
+cd Unisharp
+```
+
+Create a fresh conda environment:
+
+```bash
+conda create -n unisharp python=3.12 -y
+conda activate unisharp
+```
+
+Install PyTorch for your CUDA version. The code was smoke-tested with PyTorch 2.8 and torchvision 0.23:
+
+```bash
+pip install torch==2.8.0 torchvision==0.23.0 torchaudio==2.8.0
+```
+
+Install the remaining Python dependencies:
+
+```bash
+pip install -r requirements.txt
+```
+
+## External Dependencies
+
+### UniK3D
+
+UniSHARP uses UniK3D for universal camera ray and feature prediction. Clone the official repository into `Unisharp/UniK3D`:
+
+```bash
+git clone https://github.com/lpiccinelli-eth/UniK3D.git UniK3D
+```
+
+### 3DGEER
+
+Fisheye rendering depends on the GEER CUDA rasterizer from 3DGEER. Clone the repository into `Unisharp/3dgeer`:
+
+```bash
+git clone https://github.com/boschresearch/3dgeer.git 3dgeer
+```
+
+If you only use perspective or panoramic inference, the GEER rasterizer may not be needed. It is required for fisheye rendering paths.
+
+## Dataset
+
+The released dataset is hosted on Hugging Face:
+
+- Dataset: [Insta360-Research/OmniRooms](https://huggingface.co/datasets/Insta360-Research/OmniRooms)
+- Training manifests: [Insta360-Research/OmniRooms/manifests/train](https://huggingface.co/datasets/Insta360-Research/OmniRooms/tree/main/manifests/train)
+- Validation manifests: [Insta360-Research/OmniRooms/manifests/validation](https://huggingface.co/datasets/Insta360-Research/OmniRooms/tree/main/manifests/validation)
+
+**OmniRooms** is a panoramic simulation dataset highly suitable for 3D reconstruction, especially for 3DGS tasks. It consists of 16 large indoor scenes, each containing multiple rooms, and 300k RGB images covering both small and large pose movements with corresponding depth information. OmniRooms is collected via AirSim, with **OmniRooms-Wide** derived by projecting these panoramas into 130-degree equidistant fisheye views. For each anchor point on a 0.5 m voxel grid, one central camera and 29 randomly sampled cameras within a 0.3 m-radius sphere are rendered. To isolate translation-induced synthesis, all cameras share a fixed orientation. Each frame is rendered as a 1024 x 2048 ERP image.
+
+<p align="center">
+  <img src="assets/dataset/AIUE5_vol8_03_2x2.jpg" width="24%" alt="OmniRooms scene AIUE5 vol8 03">
+  <img src="assets/dataset/AIUE5_vol8_04_2x2.jpg" width="24%" alt="OmniRooms scene AIUE5 vol8 04">
+  <img src="assets/dataset/AIUE5_vol8_05_2x2.jpg" width="24%" alt="OmniRooms scene AIUE5 vol8 05">
+  <img src="assets/dataset/AIUE_V01_001_2x2.jpg" width="24%" alt="OmniRooms scene AIUE V01 001">
+  <br>
+  <img src="assets/dataset/AIUE_V01_003_2x2.jpg" width="24%" alt="OmniRooms scene AIUE V01 003">
+  <img src="assets/dataset/AIUE_V01_004_2x2.jpg" width="24%" alt="OmniRooms scene AIUE V01 004">
+  <img src="assets/dataset/AIUE_V02_001_2x2.jpg" width="24%" alt="OmniRooms scene AIUE V02 001">
+  <img src="assets/dataset/AI_vol3_01_2x2.jpg" width="24%" alt="OmniRooms scene AI vol3 01">
+  <br>
+  <img src="assets/dataset/AI_vol3_02_2x2.jpg" width="24%" alt="OmniRooms scene AI vol3 02">
+  <img src="assets/dataset/AI_vol3_03_2x2.jpg" width="24%" alt="OmniRooms scene AI vol3 03">
+  <img src="assets/dataset/AI_vol3_04_2x2.jpg" width="24%" alt="OmniRooms scene AI vol3 04">
+  <img src="assets/dataset/AI_vol4_01_2x2.jpg" width="24%" alt="OmniRooms scene AI vol4 01">
+  <br>
+  <img src="assets/dataset/AI_vol4_02_2x2.jpg" width="24%" alt="OmniRooms scene AI vol4 02">
+  <img src="assets/dataset/AI_vol4_03_2x2.jpg" width="24%" alt="OmniRooms scene AI vol4 03">
+  <img src="assets/dataset/AI_vol4_04_2x2.jpg" width="24%" alt="OmniRooms scene AI vol4 04">
+  <img src="assets/dataset/AI_vol4_05_2x2.jpg" width="24%" alt="OmniRooms scene AI vol4 05">
+</p>
+
+The code supports the following data sources and manifest aliases:
+
+- `RealEstate10K`
+- `HM3D`
+- `OmniRooms`
+- `OmniRooms-Wide`
+- `WildRGB-D`
+- `DL3DV` 
+- `ScanNet++ Fisheye`
+- `Replica`, and `Tanks and Temples` for validation-only protocols
+
+Training manifests use the names released under `manifests/train`:
+
+```text
+dataset_manifests/
+├── re10k_train_chunks.txt            
+├── hm3d_train_scenes.txt            
+├── omnirooms.txt              
+├── wildrgbd_train_scenes.txt         
+├── dl3dv_train_scenes.txt            
+└── scanetpp_fisheye_train_scenes.txt 
+```
+
+Validation manifests use the names released under `manifests/validation`:
+
+```text
+validation_manifests/
+├── re10k.txt                      
+├── dl3dv.txt                         
+├── hm3d.txt                          
+├── omnirooms.txt                      
+├── omnirooms_wide.txt              
+├── wildrgbd.txt                     
+├── scanetpp_fisheye.txt              
+├── replica.txt                       
+├── tat.txt                           
+```
+
+## Checkpoints
+
+Training starts UniSHARP heads from scratch and loads the original pretrained UniK3D weights through the UniK3D loader. The official launcher does not resume from a previous UniSHARP checkpoint by default.
+
+Released UniSHARP checkpoints are available at [Insta360-Research/Unisharp](https://huggingface.co/Insta360-Research/Unisharp/tree/main). Place a checkpoint anywhere on disk and pass the path to validation or inference:
+
+```bash
+CHECKPOINT=/path/to/step_XXXXXXX.pt
+```
+
+## Training
+
+Use the official gt-override training launcher:
+
+```bash
+bash scripts/train.sh
+```
+
+Training outputs are saved under:
+
+```text
+outputs/<run_name>/
+├── config.json
+├── losses.csv
+├── step_XXXXXXX.pt
+└── vis/
+```
+
+## Validation
+
+Run validation with a checkpoint:
+
+```bash
+bash scripts/validate_unisharp.sh /path/to/step_XXXXXXX.pt
+```
+
+## Inference
+
+Run single-image inference:
+
+```bash
+python scripts/infer_unisharp.py \
+  --checkpoint /path/to/step_XXXXXXX.pt \
+  --image /path/to/image.jpg \
+  --out-dir outputs/inference
+```
+
+Run a directory or image list:
+
+```bash
+python scripts/infer_unisharp.py \
+  --checkpoint /path/to/step_XXXXXXX.pt \
+  --image-dir /path/to/images \
+  --out-dir outputs/inference
+```
+
+If calibrated camera parameters are available, pass them through a JSON file. Without this file, the script predicts rays with UniK3D and fits the camera parameters automatically.
+
+Example perspective camera JSON:
+
+```json
+{
+  "camera": "perspective",
+  "intrinsics": {
+    "fx": 820.0,
+    "fy": 820.0,
+    "cx": 512.0,
+    "cy": 384.0
+  }
+}
+```
+
+```bash
+python scripts/infer_unisharp.py \
+  --checkpoint /path/to/step_XXXXXXX.pt \
+  --image /path/to/perspective.jpg \
+  --camera-json /path/to/perspective_camera.json
+```
+
+Example Fisheye624 camera JSON:
+
+```json
+{
+  "camera": "fisheye",
+  "camera_params": [820.0, 820.0, 512.0, 384.0, 0.01, -0.001, 0.0, 0.0]
+}
+```
+
+```bash
+python scripts/infer_unisharp.py \
+  --checkpoint /path/to/step_XXXXXXX.pt \
+  --image /path/to/fisheye.jpg \
+  --camera-json /path/to/fisheye_camera.json
+```
+
+For batched inference, the JSON can also contain per-image entries:
+
+```json
+{
+  "default": {
+    "camera": "perspective",
+    "intrinsics": [820.0, 820.0, 512.0, 384.0]
+  },
+  "images": {
+    "panorama.jpg": {
+      "camera": "panorama"
+    },
+    "fisheye.jpg": {
+      "camera": "fisheye",
+      "camera_params": [820.0, 820.0, 512.0, 384.0, 0.01, -0.001, 0.0, 0.0]
+    }
+  }
+}
+```
+
+
+## Acknowledgement
+
+This project builds on open-source work from:
+
+- [SHARP](https://github.com/apple/ml-sharp) for monocular Gaussian view synthesis
+- [UniK3D](https://github.com/lpiccinelli-eth/UniK3D) for universal camera geometry and features
+- [3DGEER](https://github.com/boschresearch/3dgeer) for generic-camera Gaussian rasterization
+- [gsplat](https://github.com/nerfstudio-project/gsplat) for Gaussian splatting utilities
+
+## Citation
+
+```bibtex
+@article{song2026unisharp,
+  title={UniSHARP: Universal Sharp Monocular View Synthesis},
+  author={Song, Meixi and Zhang, Dizhe and Ren, Hao and Zhang, Ruiyang and Du, Bo and Yang, Ming-Hsuan and Qi, Lu},
+  journal={arXiv},
+  year={2026}
+}
+```

requirements.txt

@@ -0,0 +1,37 @@
+torch==2.8.0
+torchvision==0.23.0
+torchaudio==2.8.0
+triton>=3.4.0
+
+gsplat==1.5.3
+ninja>=1.11.1
+
+numpy>=2.0.0
+scipy>=1.11.0
+Pillow>=10.2.0
+pillow-heif>=0.16.0
+imageio>=2.34.0
+imageio-ffmpeg>=0.5.1
+matplotlib>=3.8.0
+h5py>=3.10.0
+opencv-python>=4.9.0
+plyfile>=1.0.3
+
+click>=8.1.7
+tqdm>=4.66.0
+requests>=2.31.0
+PyYAML>=6.0.1
+rich>=13.7.0
+
+huggingface-hub>=0.22.0
+timm>=1.0.0
+safetensors>=0.4.2
+einops>=0.7.0
+protobuf>=4.25.3
+
+lpips>=0.1.4
+pandas>=2.2.0
+tables>=3.9.0
+tabulate>=0.9.0
+termcolor>=2.4.0
+trimesh>=4.3.0