README.md

---
license: apache-2.0
library_name: pytorch
datasets:
- nuscenes
tags:
- object-detection
- 3d-object-detection
- autonomous-driving
- camera-radar-fusion
- bev
- nuscenes
- pytorch
- research
- multi-task-learning
model-index:
- name: CARMA
  results:
  - task:
      type: object-detection
      name: BEV 3D object detection
    dataset:
      type: nuscenes
      name: nuScenes v1.0 trainval validation
    metrics:
    - type: custom_map_2m
      name: Custom mAP at 2 m center distance
      value: 0.4922
    - type: precision_2m
      name: Precision at 2 m center distance
      value: 0.7970
    - type: recall_2m
      name: Recall at 2 m center distance
      value: 0.5736
    - type: f1_2m
      name: F1 at 2 m center distance
      value: 0.6671
---

# CARMA

CARMA is a PyTorch checkpoint for camera and radar BEV object detection on nuScenes. The model uses six surround cameras and radar sweeps at inference time, then predicts object centers and 3D boxes in a bird's eye view representation.

This repository currently contains the trained checkpoint:

```text
CARMA_map0.4922.pth
```

The checkpoint is intended for research, dissertation experiments, qualitative visualization, and reproducibility of the CARMA detector. It is not a standalone Hugging Face Transformers model and is not configured for hosted Inference API deployment.

## Model Overview

CARMA combines image and radar information in a BEV detection pipeline:

- Camera branch: six nuScenes RGB cameras are encoded and lifted into a BEV representation.
- Radar branch: radar points and sweeps are encoded into BEV features.
- Fusion module: modality projection, deformable alignment, bidirectional cross attention, and gated residual fusion combine camera and radar features.
- Detection head: a center based heatmap predicts object centers, while regression channels decode 3D box geometry, orientation, and velocity.
- Auxiliary training heads: LiDAR derived supervision is used during training for depth, camera BEV, and related auxiliary branches. These LiDAR derived targets are not required at inference time.

Final inference inputs:

- Six camera images.
- Camera intrinsics and extrinsics.
- Radar points or radar BEV tensors.

Not required at inference time:

- LiDAR point clouds.
- LiDAR occupancy maps.
- LiDAR derived depth supervision targets.

## Classes

The checkpoint predicts the standard 10 nuScenes detection classes used by this project:

```text
car, truck, construction_vehicle, bus, trailer,
barrier, motorcycle, bicycle, pedestrian, traffic_cone
```

## Evaluation

The checkpoint was selected using the project validation metric at a 2 m center distance threshold.

Important note: the metrics below are from the CARMA project validation script. They are custom center distance metrics and should not be reported as the official nuScenes detection benchmark, NDS, or official nuScenes mAP unless the official nuScenes evaluation protocol is run separately.

### nuScenes v1.0 trainval validation

Validation configuration:

- Dataset: nuScenes v1.0 trainval validation split.
- Decode setting: raw heatmap at score threshold 0.4.
- Main matching threshold: 2 m center distance.
- Selected checkpoint: `CARMA_map0.4922.pth`.

| Metric | Value |
| --- | ---: |
| Precision at 2 m | 0.7970 |
| Recall at 2 m | 0.5736 |
| F1 at 2 m | 0.6671 |
| mAP at 0.5 m | 0.1892 |
| mAP at 1 m | 0.3573 |
| mAP at 2 m | 0.4922 |
| mAP at 4 m | 0.5548 |
| Mean localization distance | 0.526 m |
| True positives | 89969 |
| False positives | 22909 |
| False negatives | 66877 |

### Per Class Results

| Class | AP | Precision | Recall | F1 | Mean error |
| --- | ---: | ---: | ---: | ---: | ---: |
| car | 0.629 | 0.855 | 0.645 | 0.736 | 0.46 m |
| truck | 0.517 | 0.672 | 0.598 | 0.633 | 0.67 m |
| construction_vehicle | 0.389 | 0.653 | 0.444 | 0.529 | 0.83 m |
| bus | 0.560 | 0.610 | 0.679 | 0.643 | 0.88 m |
| trailer | 0.347 | 0.546 | 0.488 | 0.516 | 1.01 m |
| barrier | 0.582 | 0.830 | 0.611 | 0.704 | 0.58 m |
| motorcycle | 0.503 | 0.731 | 0.521 | 0.608 | 0.53 m |
| bicycle | 0.453 | 0.745 | 0.471 | 0.577 | 0.51 m |
| pedestrian | 0.364 | 0.779 | 0.386 | 0.516 | 0.54 m |
| traffic_cone | 0.578 | 0.761 | 0.615 | 0.680 | 0.43 m |

### nuScenes Mini Validation Sanity Check

The same checkpoint was also evaluated on the nuScenes mini validation split as a quick sanity check.

| Metric | Value |
| --- | ---: |
| Precision at 2 m | 0.7783 |
| Recall at 2 m | 0.5729 |
| F1 at 2 m | 0.6600 |
| mAP at 0.5 m | 0.2830 |
| mAP at 1 m | 0.4804 |
| mAP at 2 m | 0.5707 |
| mAP at 4 m | 0.6278 |
| Mean localization distance | 0.524 m |

## Usage

Install Git LFS before cloning the model repository:

```bash
git lfs install
git clone https://huggingface.co/AdrianNaziru/CARMA
```

The checkpoint can be inspected with PyTorch:

```python
import torch

checkpoint = torch.load("CARMA/CARMA_map0.4922.pth", map_location="cpu")
print(checkpoint.keys() if isinstance(checkpoint, dict) else type(checkpoint))
```

To run inference, use the CARMA project code that defines the model architecture, preprocessing, nuScenes dataset loading, and decoding utilities. The checkpoint file alone is not sufficient without the matching source code and configuration.

Typical local commands from the CARMA source tree:

```powershell
python realtime_bev_inference_pro_fast.py

```

Expected local data layout:

```text
data/nuscenes/v1.0-trainval
```

The main runtime settings used for this checkpoint are:

```text
NUSCENES_VERSION=v1.0-trainval
BEV grid=192 x 192
point cloud range=[-51.2, -51.2, -5.0, 51.2, 51.2, 3.0]
image size=256 x 704
radar sweeps=13
decode heat=raw
score threshold=0.4 for the reported validation metrics
```

## Limitations

- This checkpoint is for research and academic analysis only.
- It is not validated for safety critical autonomous driving deployment.
- It depends on the exact CARMA source code, nuScenes preprocessing, calibration handling, and decoding logic used during training.
- Reported metrics are custom project metrics and are not official nuScenes leaderboard metrics.
- Performance may degrade outside the nuScenes sensor setup, camera calibration, weather, geography, or object distribution.
- LiDAR supervision improves training signals, but LiDAR is not used during final inference.

## Citation

If you use this checkpoint, please cite the CARMA dissertation or the associated project repository when available.

## License

This model repository is released under the Apache 2.0 license.