Add README.md

README.md

@@ -0,0 +1,72 @@
+---
+license: other
+tags:
+  - heal
+  - horizon
+  - bev
+  - lidar
+---
+
+# BevFusion+PointPillar+HENet Multisensor Multitask
+
+The BevFusion multisensor multitask model extracts BEV features via dual branches: the camera branch uses HENet to extract multi-view features, then BevFormer ViewTransformer to BEV; the lidar branch uses PointPillars (`PillarFeatureNet` + `PointPillarScatter` + `SECONDNeck`) to voxelize point clouds and extract BEV features. Fused BEV features feed CenterPoint detection head (3D object detection) and occupancy head (semantic occupancy prediction) for joint det+occ training.
+
+---
+
+## Deployment Metrics
+
+### Model Parameters
+
+| Model | Model Input | Backbone | Neck | Model Output |
+|---|---|---|---|---|
+| BevFusion | 6-camera multi-view images `(B,6,3,512,960)` + lidar point cloud `(B,N,5)` | PointPillar (lidar) + HENet (camera) | FPN (camera) + SECONDNeck (lidar) | det bounding boxes `(B,N,cls+reg)`; occ occupancy grid `(B,C,H,W)` |
+
+### Accuracy Metrics
+
+| March | Metric | float | calibration | qat | hbm |
+| --- | --- | --- | --- | --- | --- |
+| J6M | NDS | 0.6421 | 0.6301 | — | 0.6294 |
+|  | mAP | 0.5825 | 0.5724 | — | 0.5726 |
+|  | Occ mIoU | 0.5187 | 0.52 | — | 0.5206 |
+
+> Results measured with `march = March.NASH_M` (J6M) configuration; this task has no QAT stage (qat column is `—`).
+>
+> HEAL version: heal 0.0.2 / hbdk4-compiler 4.11.11 / horizon_plugin_pytorch 3.3.10.
+
+### Performance Metrics
+
+> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.
+
+| March | latency (ms) | fps | Memory Usage |
+|---|---|---|---|
+| J6M | 23.93 | 49.41 | 187.10 |
+| J6P | 16.87 | 281.67 | 195.70 |
+| J6B | - | - | - |
+
+J6B performance is not available for this model.
+
+---
+
+## Model Overview
+
+### Core Design
+
+The BevFusion multisensor multitask model extracts BEV features via dual branches: the camera branch uses HENet to extract multi-view features, then BevFormer ViewTransformer to BEV; the lidar branch uses PointPillars (`PillarFeatureNet` + `PointPillarScatter` + `SECONDNeck`) to voxelize point clouds and extract BEV features. Fused BEV features feed CenterPoint detection head (3D object detection) and occupancy head (semantic occupancy prediction) for joint det+occ training.
+
+- **Task type**: Multisensor multitask fusion (3D object detection + occupancy grid prediction).
+- **backbone**: Camera HENet (multi-view feature extraction) + lidar PointPillars (`PillarFeatureNet` + `PointPillarScatter`).
+- **neck**: Camera FPN + lidar SECONDNeck.
+- **Detection head**: CenterPoint detection head, outputting 10-class 3D bounding boxes + velocity (`num_classes=10`).
+- **Occupancy head**: Semantic occupancy prediction, 18 classes (`num_classes_occ=18`).
+- **BEV range**: `bev_size=(51.2, 51.2, 0.8)`, `bev_size_occ=(40, 40)`; `point_cloud_range=[-51.2,-51.2,-5.0,51.2,51.2,3.0]`.
+- **Model input**: 6-camera multi-view images (B,6,3,512,960) + lidar point cloud (B,N,D).
+- **Model output**: det 3D bounding boxes + occ occupancy grid semantics.
+
+**Deployment note**: HBIR export enables `enable_vpu=True`; compilation uses `input_source="ddr, ddr, pyramid, ddr, ddr, ddr, ddr"` (DDR preferred; lidar features read from DDR).
+
+### Official Repo and Paper
+
+Official repo: https://github.com/mit-han-lab/bevfusion
+Paper: https://arxiv.org/abs/2205.13542
+
+Note: Camera branch is BevFormer, lidar branch is PointPillars/CenterPoint; camera backbone HENet is a HEAL in-house implementation; the official repo uses a different backbone.