File size: 3,349 Bytes
5449960 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | ---
license: other
tags:
- heal
- horizon
- bev
- lidar
---
# SparseBEV + Lidar Fusion + HENet-tiny
SparseBevFusion extends SparseBEV with a lidar branch: HENet-tiny extracts camera features, `CenterPointDetector` (`PillarFeatureNet` + `PointPillarScatter`) processes lidar point clouds, `DeformableFeatureAggregationLiF` (with `InstanceFuseModule`) fuses camera-lidar features in BEV space, and `SparseBEVHead` performs sparse query detection.
---
## Deployment Metrics
### Model Parameters
| Model | Model Input | Backbone | Neck | Model Output |
|---|---|---|---|---|
| SparseBevFusion | 6-camera multi-view images `(B,6,3,256,704)` + lidar point cloud `(B,N,5)` | HENet-tiny (camera) + PointPillarScatter (lidar) | MMFPN + DenseDepthNet + DFA-LiF | 3D bounding boxes `(B,N,cls+reg)` |
### Accuracy Metrics
| March | Metric | float | calibration | qat | hbm |
| --- | --- | --- | --- | --- | --- |
| J6M | NDS | 0.6704 | 0.651 | 0.6647 | 0.6628 |
| | mAP | 0.6086 | 0.5853 | 0.6076 | 0.5961 |
> Results measured with `march = March.NASH_M` (J6M) configuration.
>
> HEAL version: heal 0.0.2 / hbdk4-compiler 4.11.11 / horizon_plugin_pytorch 3.3.10.
### Performance Metrics
> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.
| March | latency (ms) | fps | Memory Usage |
|---|---|---|---|
| J6M | 22.01 | 55.91 | 151.80 |
| J6P | 16.55 | 309.27 | 162.90 |
| J6B | 85.92 | 19.14 | 84.00 |
---
## Model Overview
### Core Design
SparseBevFusion extends SparseBEV with a lidar branch: HENet-tiny extracts camera features, `CenterPointDetector` (`PillarFeatureNet` + `PointPillarScatter`) processes lidar point clouds, `DeformableFeatureAggregationLiF` (with `InstanceFuseModule`) fuses camera-lidar features in BEV space, and `SparseBEVHead` performs sparse query detection.
- **Task type**: BEV 3D object detection (BEV 3D Object Detection, camera + lidar fusion).
- **backbone**: HENet-tiny (`type=HENet`, `in_channels=3`, `embed_dims=[64,128,192,384]`, multi-view camera feature extraction); lidar branch `CenterPointDetector` with `PillarFeatureNet` (`num_input_features=5`) + `PointPillarScatter` + HENet (`in_channels=64`) for pillar features (`voxel_size=[0.2,0.2,8]`).
- **neck**: `MMFPN` (camera branch, `in_strides=[2,4,8,16,32]`→`out_strides=[4,8,16,32]`) + `DenseDepthNet` (dense depth auxiliary) + `DeformableFeatureAggregationLiF` (with `InstanceFuseModule`, BEV camera-lidar feature fusion).
- **Detection head**: `SparseBEVHead` (`MemoryBank` + `SparseBEVEncoder`, sparse query + `DeformableFeatureAggregationLiF` fusion, `num_classes=10`, `num_decoder=6`, `num_anchors=384`).
- **Loss function**: `FocalLoss` (cls) + `L1Loss` (reg) + `CrossEntropyLoss` (cns) + `GaussianFocalLoss` (yns).
- **Model input**: 6-camera multi-view images `(B,6,3,256,704)` + lidar point cloud `(B,N,5)` (`load_dim=5`, `use_dim=[0,1,2,3,4]`, `num_lidar_sweeps=9`, `voxel_size=[0.2,0.2,8]`, `max_voxels=(30000,40000)`).
- **Model output**: 10-class 3D bounding boxes (`num_classes=10`), `(B,N,cls+reg)`.
### Official Repo and Paper
Official repo: https://github.com/yichen928/SparseFusion
Paper: https://arxiv.org/abs/2304.14340
Note: The camera backbone HENet is a HEAL in-house implementation; the official repo uses a different backbone.
|