SiMO: Single-Modal-Operable Multimodal Collaborative Perception
This repository contains pretrained checkpoints for SiMO (Single-Modal-Operable Multimodal Collaborative Perception), a novel framework for robust multimodal collaborative 3D object detection in autonomous driving.
📜 Paper
Title: Single-Modal-Operable Multimodal Collaborative Perception Conference: ICLR 2026 OpenReview: Link
🚀 Key Features
- Single-Modal Operability: Maintains functional performance when one modality fails
- LAMMA Fusion: Length-Adaptive Multi-Modal Fusion module
- PAFR Training: Pretrain-Align-Fuse-Random Drop training strategy
- Graceful Degradation: >80% AP@30 with camera-only operation
📦 Available Models
| Model | Dataset | Architecture | Checkpoint |
|---|---|---|---|
| SiMO-PF | OPV2V-H | Pyramid Fusion + LAMMA | Download |
| SiMO-AttFuse | OPV2V-H | AttFusion + LAMMA | Download |
📊 Performance
OPV2V-H (with Random Drop)
| Modality | AP@30 | AP@50 | AP@70 |
|---|---|---|---|
| LiDAR + Camera | 98.30 | 97.94 | 94.64 |
| LiDAR-only | 97.32 | 97.07 | 94.06 |
| Camera-only | 80.81 | 69.63 | 44.82 |
💻 Usage
Installation
git clone https://github.com/dempsey-wen/SiMO.git
cd SiMO
pip install -r requirements.txt
Download Checkpoint
# Install huggingface-hub
pip install huggingface-hub
# Download specific checkpoint
python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='DempseyWen/SiMO', filename='***.pth')"
📖 Full Documentation
For complete documentation, training scripts, and data preparation instructions, please visit our GitHub repository.
🏢 Acknowledgements
This work builds upon:
📄 Citation
If you find this work useful, please cite:
@inproceedings{wen2026simo,
title={Single-Modal-Operable Multimodal Collaborative Perception},
author={Wen, Dempsey and Lu, Yifan and others},
booktitle={International Conference on Learning Representations (ICLR)},
year={2026}
}
📄 License
MIT License - see LICENSE for details.