---
license: other
tags:
  - heal
  - horizon
  - bev
  - lidar
---

# BevFusion+PointPillar+HENet Multisensor Multitask

The BevFusion multisensor multitask model extracts BEV features via dual branches: the camera branch uses HENet to extract multi-view features, then BevFormer ViewTransformer to BEV; the lidar branch uses PointPillars (`PillarFeatureNet` + `PointPillarScatter` + `SECONDNeck`) to voxelize point clouds and extract BEV features. Fused BEV features feed CenterPoint detection head (3D object detection) and occupancy head (semantic occupancy prediction) for joint det+occ training.

---

## Deployment Metrics

### Model Parameters

| Model | Model Input | Backbone | Neck | Model Output |
|---|---|---|---|---|
| BevFusion | 6-camera multi-view images `(B,6,3,512,960)` + lidar point cloud `(B,N,5)` | PointPillar (lidar) + HENet (camera) | FPN (camera) + SECONDNeck (lidar) | det bounding boxes `(B,N,cls+reg)`; occ occupancy grid `(B,C,H,W)` |

### Accuracy Metrics

| March | Metric | float | calibration | qat | hbm |
| --- | --- | --- | --- | --- | --- |
| J6M | NDS | 0.6421 | 0.6301 | — | 0.6294 |
|  | mAP | 0.5825 | 0.5724 | — | 0.5726 |
|  | Occ mIoU | 0.5187 | 0.52 | — | 0.5206 |

> Results measured with `march = March.NASH_M` (J6M) configuration; this task has no QAT stage (qat column is `—`).
>
> HEAL version: heal 0.0.2 / hbdk4-compiler 4.11.11 / horizon_plugin_pytorch 3.3.10.

### Performance Metrics

> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.

| March | latency (ms) | fps | Memory Usage |
|---|---|---|---|
| J6M | 23.93 | 49.41 | 187.10 |
| J6P | 16.87 | 281.67 | 195.70 |
| J6B | - | - | - |

J6B performance is not available for this model.

---

## Model Overview

### Core Design

The BevFusion multisensor multitask model extracts BEV features via dual branches: the camera branch uses HENet to extract multi-view features, then BevFormer ViewTransformer to BEV; the lidar branch uses PointPillars (`PillarFeatureNet` + `PointPillarScatter` + `SECONDNeck`) to voxelize point clouds and extract BEV features. Fused BEV features feed CenterPoint detection head (3D object detection) and occupancy head (semantic occupancy prediction) for joint det+occ training.

- **Task type**: Multisensor multitask fusion (3D object detection + occupancy grid prediction).
- **backbone**: Camera HENet (multi-view feature extraction) + lidar PointPillars (`PillarFeatureNet` + `PointPillarScatter`).
- **neck**: Camera FPN + lidar SECONDNeck.
- **Detection head**: CenterPoint detection head, outputting 10-class 3D bounding boxes + velocity (`num_classes=10`).
- **Occupancy head**: Semantic occupancy prediction, 18 classes (`num_classes_occ=18`).
- **BEV range**: `bev_size=(51.2, 51.2, 0.8)`, `bev_size_occ=(40, 40)`; `point_cloud_range=[-51.2,-51.2,-5.0,51.2,51.2,3.0]`.
- **Model input**: 6-camera multi-view images (B,6,3,512,960) + lidar point cloud (B,N,D).
- **Model output**: det 3D bounding boxes + occ occupancy grid semantics.

**Deployment note**: HBIR export enables `enable_vpu=True`; compilation uses `input_source="ddr, ddr, pyramid, ddr, ddr, ddr, ddr"` (DDR preferred; lidar features read from DDR).

### Official Repo and Paper

Official repo: https://github.com/mit-han-lab/bevfusion
Paper: https://arxiv.org/abs/2205.13542

Note: Camera branch is BevFormer, lidar branch is PointPillars/CenterPoint; camera backbone HENet is a HEAL in-house implementation; the official repo uses a different backbone.

### Reference

For more J6 chip deployment details, see https://developer.horizon.auto/blog/14092