InSAI-Lab/360BEV

360Mapper: Panoramic Semantic Mapping for Indoor Bird's-Eye View

[IEEE/CVF WACV 2024] Official Model Checkpoints Repository for 360BEV and 360Mapper.

• GitHub Repository: https://github.com/InSAI-Lab/360BEV
• Project Page: https://jamycheung.github.io/360BEV.html
• Dataset Repository: InSAI-Lab/360BEV
• Paper: arXiv:2303.11910
• Collection: 360BEV & 360Mapper (WACV 2024)

---

Model Description

360Mapper is designed for end-to-end panoramic semantic mapping to indoor Bird's-Eye View (BEV). It bridges 360° panoramic observations with top-down allocentric metric maps using cross-attention mechanisms, deformable attention, and high-capacity backbones (SegFormer MiT and SegNeXt MSCAN).

Benchmark Performance

1. 360FV Stanford-2D3D (Front-View Semantic Segmentation)

Model	Backbone	Input	mIoU (%)	Checkpoint
Trans4PASS	MiT-B2	RGB	52.10	--
CBFC	ResNet-101	RGB	52.20	--
Ours (360FV)	MiT-B2	RGB	54.30	360FV_Stanford/

2. 360FV Matterport3D (Front-View Semantic Segmentation)

Model	Backbone	Input	mIoU (%)	Checkpoint
HoHoNet	ResNet-101	RGB-D	44.85	--
SegFormer	MiT-B2	RGB	45.53	--
Ours (360FV)	MiT-B2	RGB	46.35	360FV_Matterport/fv_mp3d.pkl

3. 360BEV-Stanford (Panoramic BEV Semantic Mapping)

Method	Backbone	Acc (%)	mRecall (%)	mPrecision (%)	mIoU (%)	Checkpoint
Trans4Map	MiT-B0	86.41	40.45	57.47	32.26	--
Trans4Map	MiT-B2	86.53	45.28	62.61	36.08	--
360Mapper (Ours)	MiT-B0	92.07	50.14	65.37	42.42	360BEV_Stanford/360BEV_s2d3d_segformer_b0.pkl
360Mapper (Ours)	MiT-B2	92.80	53.56	67.72	45.78	360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl
360Mapper (Ours)	MSCA-B	92.67	55.02	68.02	46.44	360BEV_Stanford/360BEV_s2d3d_segnext_B.pkl

4. 360BEV-Matterport (Panoramic BEV Semantic Mapping)

Method	Backbone	Acc (%)	mRecall (%)	mPrecision (%)	mIoU (%)	Checkpoint
Trans4Map	MiT-B0	70.19	44.31	50.39	31.92	--
Trans4Map	MiT-B2	73.28	51.60	53.02	36.72	--
360Mapper (Ours)	MiT-B0	75.44	48.80	56.01	36.98	360BEV_Matterport/360BEV_mp3d_segformer_b0.pkl
360Mapper (Ours)	MiT-B2	78.80	59.54	59.97	44.32	360BEV_Matterport/360BEV_mp3d_segformer_b2.pkl
360Mapper (Ours)	MSCA-B	78.93	60.51	62.83	46.31	360BEV_Matterport/360BEV_mp3d_segnext_B.pkl

Checkpoints Organization

├── 360FV_Stanford/
│   ├── fv_s2d3d_fold1.pkl
│   ├── fv_s2d3d_fold2.pkl
│   └── fv_s2d3d_fold3.pkl
├── 360FV_Matterport/
│   └── fv_mp3d.pkl
├── 360BEV_Stanford/
│   ├── 360BEV_s2d3d_segformer_b0.pkl
│   ├── 360BEV_s2d3d_segformer_b2.pkl
│   └── 360BEV_s2d3d_segnext_B.pkl
└── 360BEV_Matterport/
    ├── 360BEV_mp3d_segformer_b0.pkl
    ├── 360BEV_mp3d_segformer_b2.pkl
    └── 360BEV_mp3d_segnext_B.pkl

Quick Start & Evaluation

1. Download Weights

Using Hugging Face CLI:

# Download all checkpoints
hf download InSAI-Lab/360BEV --local-dir ./checkpoints

# Or download specific model
hf download InSAI-Lab/360BEV 360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl --local-dir ./checkpoints

Using Python:

from huggingface_hub import hf_hub_download

model_path = hf_hub_download(
    repo_id="InSAI-Lab/360BEV",
    filename="360BEV_Stanford/360BEV_s2d3d_segformer_b2.pkl",
    local_dir="./checkpoints"
)

2. Evaluation

In the 360BEV codebase:

# 360BEV Matterport
python test_360BEV_Matterport.py --config configs/model_360BEV_mp3d.yml

# 360BEV Stanford-2D3D
python test_360BEV_S2d3d.py --config configs/model_360BEV_s2d3d.yml

# Stanford2D3D (360FV)
python test_pano_360Attention_S2d3d.py --config configs/model_fv_s2d3d.yml

# 360FV-Matterport
python test_pano_360Attention_Matterport.py --config configs/model_fv_mp3d.yml

Citation

@inproceedings{teng2024_360bev,
  title={360BEV: Panoramic Semantic Mapping for Indoor Bird's-Eye View}, 
  author={Teng, Zhifeng and Zhang, Jiaming and Yang, Kailun and Peng, Kunyu and Shi, Hao and Reiß, Simon and Cao, Ke and Stiefelhagen, Rainer},
  booktitle={2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)},
  year={2024}
}

License

This repository is licensed under the Apache-2.0 License.