README.md

---
license: apache-2.0
pipeline_tag: object-detection
tags:
  - autoware
  - ros2
  - autonomous-driving
  - lidar
  - camera
  - point-cloud
  - 3d-object-detection
  - bevfusion
  - tensorrt
  - onnx
---

# BEVFusion for Autoware (`bevfusion`)

3D object detection models for LiDAR-only and camera-LiDAR fusion, used by the
[`autoware_bevfusion`](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_bevfusion)
node in [Autoware](https://github.com/autowarefoundation/autoware).

The models follow the **BEVFusion** [1] architecture (MIT Han Lab) and run with TensorRT inside Autoware.
They are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from
the ONNX files on first launch. The sparse convolution backend corresponds to
[spconv](https://github.com/traveller59/spconv), executed at inference time through the
[spconv_cpp](https://github.com/autowarefoundation/spconv_cpp) TensorRT plugins that Autoware installs
automatically in its setup script.

## Model overview

| | |
| --- | --- |
| Task | 3D object detection (oriented bounding boxes) from a LiDAR point cloud, optionally fused with multi-camera images |
| Architecture | BEVFusion (sparse-convolution LiDAR encoder, optional camera-to-BEV branch, transformer detection head) |
| Variant family | `t4base_120m` |
| Detected classes | `CAR`, `TRUCK`, `BUS`, `BICYCLE`, `PEDESTRIAN` |
| Runtime | TensorRT (FP16 by default) via the `autoware_bevfusion` ROS 2 node, with `autoware_tensorrt_plugins` for sparse convolution |
| Format | ONNX (Autoware builds the TensorRT engines locally on first launch) |
| License | Apache-2.0 |

## Variants in this repository

| Variant | Modality | ONNX files used | Cameras |
| --- | --- | --- | --- |
| `bevfusion_lidar` | LiDAR only | `bevfusion_lidar.onnx` | 0 |
| `bevfusion_camera_lidar` | Camera-LiDAR fusion | `bevfusion_camera_lidar.onnx` + `bevfusion_image_backbone.onnx` | 6 (raw 1440x1080, ROI 576x384) |

Common model parameters for both variants (from the `ml_package_*.param.yaml` files): point cloud range
`[-122.4, -122.4, -3.0, 122.4, 122.4, 5.0]` m (roughly 120 m detection radius, matching the `t4base_120m`
family name), voxel size `[0.17, 0.17, 0.2]` m, `max_points_per_voxel: 10`, `num_proposals: 500`,
`out_size_factor: 8`, `use_intensity: false`.

Pre-processing (voxelization, multi-frame densification, optional image undistortion) and post-processing
(circle NMS, IoU NMS, yaw normalization, distance-based score thresholding, area-based class remapping) run in
the node, not in the ONNX graphs.

## Files

| File | Description |
| --- | --- |
| `bevfusion_lidar.onnx` | Main network, `bevfusion_lidar` variant |
| `bevfusion_camera_lidar.onnx` | Main network, `bevfusion_camera_lidar` variant |
| `bevfusion_image_backbone.onnx` | Image backbone, used by `bevfusion_camera_lidar` |
| `ml_package_bevfusion_lidar.param.yaml` | Model parameters for `bevfusion_lidar` |
| `ml_package_bevfusion_camera_lidar.param.yaml` | Model parameters for `bevfusion_camera_lidar` |
| `detection_class_remapper.param.yaml` | Area-based class remapping (e.g. large car to truck/trailer) |
| `deploy_metadata.yaml` | Deployment metadata recording the artifact version of this repository |

> **TensorRT engines are not distributed here.** TensorRT engines are specific to the GPU architecture and
> TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files
> on first launch (or via `build_only:=true`).

## Inputs and outputs (as used by the node)

**Inputs**

| Topic | Type | Description |
| --- | --- | --- |
| `~/input/pointcloud` | `sensor_msgs/msg/PointCloud2` | Input point cloud, `PointXYZIRC` layout as defined in `autoware_point_types` |
| `~/input/image*` | `sensor_msgs/msg/Image` | Input images (RGB8), camera-lidar variant only |
| `~/input/camera_info*` | `sensor_msgs/msg/CameraInfo` | Camera intrinsics, camera-lidar variant only |

**Output** is `~/output/objects` (`autoware_perception_msgs/msg/DetectedObjects`): oriented 3D boxes with class
and score. The node also publishes per-stage processing-time debug topics.

## Usage in Autoware

The node expects these artifacts in `$HOME/autoware_data/ml_models/bevfusion/` and launches with, e.g.:

```bash
ros2 launch autoware_bevfusion bevfusion.launch.xml \
  model_name:=bevfusion_lidar \
  model_path:=$HOME/autoware_data/ml_models/bevfusion
```

`model_name` selects the variant (`bevfusion_lidar`, the default, or `bevfusion_camera_lidar`). Add
`build_only:=true` to build the TensorRT engines from the ONNX files as a one-off pre-task.
See the [package README](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_bevfusion)
for the full parameter reference.

## Training

The models were trained on TIER IV's internal database; the training data is not publicly available. The
consuming package README documents training on roughly 35k LiDAR frames for 30 epochs. Version-specific
training details for this `t4base_120m/v2` release are not publicly documented.

Related training and inference resources:

- Sparse convolution: <https://github.com/traveller59/spconv>
- Sparse convolution TensorRT plugins used by Autoware: <https://github.com/autowarefoundation/spconv_cpp>

## Limitations

- Trained on TIER IV's internal sensor configurations; accuracy on a different LiDAR or camera setup (mounting
  positions, beam count, camera count and resolution) can drop without fine-tuning.
- Only the five classes above are detected. Other road users fall outside the label set.
- The input point cloud must follow the `PointXYZIRC` layout defined in `autoware_point_types`.
- The consuming package notes that full integration of the camera-LiDAR fusion mode into the Autoware pipeline
  is still future work; the model can be employed without changes as a LiDAR-only detector.

## Provenance

| | |
| --- | --- |
| Original source | `https://awf.ml.dev.web.auto/perception/models/bevfusion/t4base_120m/v2/` |
| Hugging Face tag | `v2.0` |

## Citation

```bibtex
@inproceedings{liu2023bevfusion,
  title     = {BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation},
  author    = {Liu, Zhijian and Tang, Haotian and Amini, Alexander and Yang, Xinyu and Mao, Huizi and Rus, Daniela and Han, Song},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  year      = {2023}
}
```

## References

- [1] Liu et al., "BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation", ICRA 2023, arXiv:2205.13542.
- [2] spconv, sparse convolution library: <https://github.com/traveller59/spconv>
- [3] spconv_cpp, Autoware's sparse convolution TensorRT plugin implementation: <https://github.com/autowarefoundation/spconv_cpp>