Improve model card: add paper links, authors, and usage instructions

#1
by nielsr HF Staff - opened
Files changed (1) hide show
  1. README.md +28 -23
README.md CHANGED
@@ -1,5 +1,8 @@
1
  ---
 
 
2
  license: mit
 
3
  tags:
4
  - computer-vision
5
  - 3d-object-detection
@@ -11,27 +14,21 @@ tags:
11
  - opv2v
12
  - v2xset
13
  - dair-v2x
14
- language:
15
- - en
16
- pipeline_tag: object-detection
17
  ---
18
 
19
- # SiMO: Single-Modal-Operable Multimodal Collaborative Perception
20
 
21
- This repository contains pretrained checkpoints for **SiMO** (Single-Modal-Operable Multimodal Collaborative Perception), a novel framework for robust multimodal collaborative 3D object detection in autonomous driving.
22
 
23
- ## πŸ“œ Paper
24
 
25
- **Title**: Single-Modal-Operable Multimodal Collaborative Perception
26
- **Conference**: ICLR 2026
27
- **OpenReview**: [Link](https://openreview.net/forum?id=h0iRgjTmVs)
28
 
29
- ## πŸš€ Key Features
30
 
31
- - **Single-Modal Operability**: Maintains functional performance when one modality fails
32
- - **LAMMA Fusion**: Length-Adaptive Multi-Modal Fusion module
33
- - **PAFR Training**: Pretrain-Align-Fuse-Random Drop training strategy
34
- - **Graceful Degradation**: >80% AP@30 with camera-only operation
35
 
36
  ## πŸ“¦ Available Models
37
 
@@ -58,6 +55,7 @@ This repository contains pretrained checkpoints for **SiMO** (Single-Modal-Opera
58
  git clone https://github.com/dempsey-wen/SiMO.git
59
  cd SiMO
60
  pip install -r requirements.txt
 
61
  ```
62
 
63
  ### Download Checkpoint
@@ -67,27 +65,34 @@ pip install -r requirements.txt
67
  pip install huggingface-hub
68
 
69
  # Download specific checkpoint
70
- python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='DempseyWen/SiMO', filename='***.pth')"
71
  ```
72
 
 
 
 
 
 
 
 
 
 
 
 
73
 
74
  ## πŸ“– Full Documentation
75
 
76
- For complete documentation, training scripts, and data preparation instructions, please visit our [GitHub repository](https://github.com/dempsey-wen/SiMO).
77
 
78
  ## 🏒 Acknowledgements
79
 
80
- This work builds upon:
81
- - [OpenCOOD](https://github.com/DerrickXuNu/OpenCOOD)
82
- - [HEAL](https://github.com/yifanlu0227/HEAL)
83
 
84
  ## πŸ“„ Citation
85
 
86
- If you find this work useful, please cite:
87
-
88
  ```bibtex
89
  @inproceedings{wen2026simo,
90
- title={Single-Modal-Operable Multimodal Collaborative Perception},
91
  author={Wen, Dempsey and Lu, Yifan and others},
92
  booktitle={International Conference on Learning Representations (ICLR)},
93
  year={2026}
@@ -96,4 +101,4 @@ If you find this work useful, please cite:
96
 
97
  ## πŸ“„ License
98
 
99
- MIT License - see [LICENSE](https://github.com/dempsey-wen/SiMO/blob/main/LICENSE) for details.
 
1
  ---
2
+ language:
3
+ - en
4
  license: mit
5
+ pipeline_tag: object-detection
6
  tags:
7
  - computer-vision
8
  - 3d-object-detection
 
14
  - opv2v
15
  - v2xset
16
  - dair-v2x
 
 
 
17
  ---
18
 
19
+ # SiMO: Single-Modality-Operable Multimodal Collaborative Perception
20
 
21
+ Official PyTorch implementation of **SiMO**, a framework for robust multimodal collaborative 3D object detection that remains functional even when a key sensor (like LiDAR) fails.
22
 
23
+ [[Paper](https://arxiv.org/abs/2603.08240)] [[OpenReview](https://openreview.net/forum?id=h0iRgjTmVs)] [[GitHub](https://github.com/dempsey-wen/SiMO)]
24
 
25
+ **Authors**: Dempsey Wen, Yifan Lu, and others.
 
 
26
 
27
+ ## πŸš€ Introduction
28
 
29
+ Existing multimodal collaborative perception approaches often fail when a primary modality (like LiDAR) is unavailable due to semantic mismatches. SiMO addresses this through:
30
+ - **LAMMA (Length-Adaptive Multi-Modal Fusion)**: A module that adaptively handles available modalities.
31
+ - **PAFR Training Strategy**: A four-stage paradigm (Pretrain-Align-Fuse-Random Drop) that prevents modality competition and ensures branch independence.
 
32
 
33
  ## πŸ“¦ Available Models
34
 
 
55
  git clone https://github.com/dempsey-wen/SiMO.git
56
  cd SiMO
57
  pip install -r requirements.txt
58
+ pip install -e .
59
  ```
60
 
61
  ### Download Checkpoint
 
65
  pip install huggingface-hub
66
 
67
  # Download specific checkpoint
68
+ python -c "from huggingface_hub import hf_hub_download; hf_hub_download(repo_id='DempseyWen/SiMO', filename='SiMO_PF/net_epoch27.pth')"
69
  ```
70
 
71
+ ### Inference
72
+
73
+ To run inference on a trained model (e.g., LiDAR+Camera):
74
+
75
+ ```bash
76
+ python opencood/tools/inference.py \
77
+ --model_dir saved_models/opv2v_lidarcamera_lamma3_pyramid_fusion/ \
78
+ --fusion_method intermediate
79
+ ```
80
+
81
+ For single-modality testing, modify the config `model.args.single_modality` to `lidar` or `camera` before running the inference script.
82
 
83
  ## πŸ“– Full Documentation
84
 
85
+ For complete documentation, training scripts, and data preparation instructions, please visit the [GitHub repository](https://github.com/dempsey-wen/SiMO).
86
 
87
  ## 🏒 Acknowledgements
88
 
89
+ This work builds upon [OpenCOOD](https://github.com/DerrickXuNu/OpenCOOD) and [HEAL](https://github.com/yifanlu0227/HEAL).
 
 
90
 
91
  ## πŸ“„ Citation
92
 
 
 
93
  ```bibtex
94
  @inproceedings{wen2026simo,
95
+ title={Single-Modality-Operable Multimodal Collaborative Perception},
96
  author={Wen, Dempsey and Lu, Yifan and others},
97
  booktitle={International Conference on Learning Representations (ICLR)},
98
  year={2026}
 
101
 
102
  ## πŸ“„ License
103
 
104
+ This project is licensed under the MIT License - see the [LICENSE](https://github.com/dempsey-wen/SiMO/blob/main/LICENSE) file for details.