| --- |
| license: apache-2.0 |
| library_name: pytorch |
| datasets: |
| - nuscenes |
| tags: |
| - object-detection |
| - 3d-object-detection |
| - autonomous-driving |
| - camera-radar-fusion |
| - bev |
| - nuscenes |
| - pytorch |
| - research |
| - multi-task-learning |
| model-index: |
| - name: CARMA |
| results: |
| - task: |
| type: object-detection |
| name: BEV 3D object detection |
| dataset: |
| type: nuscenes |
| name: nuScenes v1.0 trainval validation |
| metrics: |
| - type: custom_map_2m |
| name: Custom mAP at 2 m center distance |
| value: 0.4922 |
| - type: precision_2m |
| name: Precision at 2 m center distance |
| value: 0.7970 |
| - type: recall_2m |
| name: Recall at 2 m center distance |
| value: 0.5736 |
| - type: f1_2m |
| name: F1 at 2 m center distance |
| value: 0.6671 |
| --- |
| |
| # CARMA |
|
|
| CARMA is a PyTorch checkpoint for camera and radar BEV object detection on nuScenes. The model uses six surround cameras and radar sweeps at inference time, then predicts object centers and 3D boxes in a bird's eye view representation. |
|
|
| This repository currently contains the trained checkpoint: |
|
|
| ```text |
| CARMA_map0.4922.pth |
| ``` |
|
|
| The checkpoint is intended for research, dissertation experiments, qualitative visualization, and reproducibility of the CARMA detector. It is not a standalone Hugging Face Transformers model and is not configured for hosted Inference API deployment. |
|
|
| ## Model Overview |
|
|
| CARMA combines image and radar information in a BEV detection pipeline: |
|
|
| - Camera branch: six nuScenes RGB cameras are encoded and lifted into a BEV representation. |
| - Radar branch: radar points and sweeps are encoded into BEV features. |
| - Fusion module: modality projection, deformable alignment, bidirectional cross attention, and gated residual fusion combine camera and radar features. |
| - Detection head: a center based heatmap predicts object centers, while regression channels decode 3D box geometry, orientation, and velocity. |
| - Auxiliary training heads: LiDAR derived supervision is used during training for depth, camera BEV, and related auxiliary branches. These LiDAR derived targets are not required at inference time. |
|
|
| Final inference inputs: |
|
|
| - Six camera images. |
| - Camera intrinsics and extrinsics. |
| - Radar points or radar BEV tensors. |
|
|
| Not required at inference time: |
|
|
| - LiDAR point clouds. |
| - LiDAR occupancy maps. |
| - LiDAR derived depth supervision targets. |
|
|
| ## Classes |
|
|
| The checkpoint predicts the standard 10 nuScenes detection classes used by this project: |
|
|
| ```text |
| car, truck, construction_vehicle, bus, trailer, |
| barrier, motorcycle, bicycle, pedestrian, traffic_cone |
| ``` |
|
|
| ## Evaluation |
|
|
| The checkpoint was selected using the project validation metric at a 2 m center distance threshold. |
|
|
| Important note: the metrics below are from the CARMA project validation script. They are custom center distance metrics and should not be reported as the official nuScenes detection benchmark, NDS, or official nuScenes mAP unless the official nuScenes evaluation protocol is run separately. |
|
|
| ### nuScenes v1.0 trainval validation |
|
|
| Validation configuration: |
|
|
| - Dataset: nuScenes v1.0 trainval validation split. |
| - Decode setting: raw heatmap at score threshold 0.4. |
| - Main matching threshold: 2 m center distance. |
| - Selected checkpoint: `CARMA_map0.4922.pth`. |
|
|
| | Metric | Value | |
| | --- | ---: | |
| | Precision at 2 m | 0.7970 | |
| | Recall at 2 m | 0.5736 | |
| | F1 at 2 m | 0.6671 | |
| | mAP at 0.5 m | 0.1892 | |
| | mAP at 1 m | 0.3573 | |
| | mAP at 2 m | 0.4922 | |
| | mAP at 4 m | 0.5548 | |
| | Mean localization distance | 0.526 m | |
| | True positives | 89969 | |
| | False positives | 22909 | |
| | False negatives | 66877 | |
|
|
| ### Per Class Results |
|
|
| | Class | AP | Precision | Recall | F1 | Mean error | |
| | --- | ---: | ---: | ---: | ---: | ---: | |
| | car | 0.629 | 0.855 | 0.645 | 0.736 | 0.46 m | |
| | truck | 0.517 | 0.672 | 0.598 | 0.633 | 0.67 m | |
| | construction_vehicle | 0.389 | 0.653 | 0.444 | 0.529 | 0.83 m | |
| | bus | 0.560 | 0.610 | 0.679 | 0.643 | 0.88 m | |
| | trailer | 0.347 | 0.546 | 0.488 | 0.516 | 1.01 m | |
| | barrier | 0.582 | 0.830 | 0.611 | 0.704 | 0.58 m | |
| | motorcycle | 0.503 | 0.731 | 0.521 | 0.608 | 0.53 m | |
| | bicycle | 0.453 | 0.745 | 0.471 | 0.577 | 0.51 m | |
| | pedestrian | 0.364 | 0.779 | 0.386 | 0.516 | 0.54 m | |
| | traffic_cone | 0.578 | 0.761 | 0.615 | 0.680 | 0.43 m | |
|
|
| ### nuScenes Mini Validation Sanity Check |
|
|
| The same checkpoint was also evaluated on the nuScenes mini validation split as a quick sanity check. |
|
|
| | Metric | Value | |
| | --- | ---: | |
| | Precision at 2 m | 0.7783 | |
| | Recall at 2 m | 0.5729 | |
| | F1 at 2 m | 0.6600 | |
| | mAP at 0.5 m | 0.2830 | |
| | mAP at 1 m | 0.4804 | |
| | mAP at 2 m | 0.5707 | |
| | mAP at 4 m | 0.6278 | |
| | Mean localization distance | 0.524 m | |
|
|
| ## Usage |
|
|
| Install Git LFS before cloning the model repository: |
|
|
| ```bash |
| git lfs install |
| git clone https://huggingface.co/AdrianNaziru/CARMA |
| ``` |
|
|
| The checkpoint can be inspected with PyTorch: |
|
|
| ```python |
| import torch |
| |
| checkpoint = torch.load("CARMA/CARMA_map0.4922.pth", map_location="cpu") |
| print(checkpoint.keys() if isinstance(checkpoint, dict) else type(checkpoint)) |
| ``` |
|
|
| To run inference, use the CARMA project code that defines the model architecture, preprocessing, nuScenes dataset loading, and decoding utilities. The checkpoint file alone is not sufficient without the matching source code and configuration. |
|
|
| Typical local commands from the CARMA source tree: |
|
|
| ```powershell |
| python realtime_bev_inference_pro_fast.py |
| |
| ``` |
|
|
| Expected local data layout: |
|
|
| ```text |
| data/nuscenes/v1.0-trainval |
| ``` |
|
|
| The main runtime settings used for this checkpoint are: |
|
|
| ```text |
| NUSCENES_VERSION=v1.0-trainval |
| BEV grid=192 x 192 |
| point cloud range=[-51.2, -51.2, -5.0, 51.2, 51.2, 3.0] |
| image size=256 x 704 |
| radar sweeps=13 |
| decode heat=raw |
| score threshold=0.4 for the reported validation metrics |
| ``` |
|
|
| ## Limitations |
|
|
| - This checkpoint is for research and academic analysis only. |
| - It is not validated for safety critical autonomous driving deployment. |
| - It depends on the exact CARMA source code, nuScenes preprocessing, calibration handling, and decoding logic used during training. |
| - Reported metrics are custom project metrics and are not official nuScenes leaderboard metrics. |
| - Performance may degrade outside the nuScenes sensor setup, camera calibration, weather, geography, or object distribution. |
| - LiDAR supervision improves training signals, but LiDAR is not used during final inference. |
|
|
| ## Citation |
|
|
| If you use this checkpoint, please cite the CARMA dissertation or the associated project repository when available. |
|
|
| ## License |
|
|
| This model repository is released under the Apache 2.0 license. |
|
|
