| --- |
| license: other |
| tags: |
| - heal |
| - horizon |
| - bev |
| --- |
| |
| # BEV Multitask + HENet-tiny |
|
|
| `bev_multitask_henet_tinym` is a camera-only BEV multitask perception model: HENet-tiny backbone + single-frame BevFormer view transformer, jointly performing five tasks — dynamic object 3D detection (with tracking), static object 3D detection, occupancy prediction, discrete map (stop line) detection, and online vectorized map (MapTR). It covers the full pipeline from 3-stage float training → quantization calibration → HBIR export → HBM compilation → inference. |
|
|
| --- |
|
|
| ## Deployment Metrics |
|
|
| ### Model Parameters |
|
|
| | Model | Model Input | Backbone | Neck | Model Output | |
| |---|---|---|---|---| |
| | BEVFusionMultitask | 6-camera multi-view images `(B,6,3,512,960)` | HENet-tiny | FPN + SingleBevFormerViewTransformer | dynamic/static 3D boxes `(B,N,cls+reg)`; occ voxel semantics; discrete-map 2D boxes; map polylines | |
|
|
| ### Accuracy Metrics |
|
|
| | March | Metric | float | calibration | qat | hbm | |
| | --- | --- | --- | --- | --- | --- | |
| | J6M | — | — | — | — | — | |
|
|
| > Accuracy numbers not yet measured; fill in after evaluation. Results use `march = March.NASH_M` (J6M) configuration; this task has no QAT stage (qat column is `—`). |
| > |
| > HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.1 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2). |
| |
| ### Performance Metrics |
| |
| > **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage. |
| > Data source: `tools/test_perf_all.py` (default sampling interval `-d 200ms`, perf-time 1). |
| |
| | March | latency (ms) | fps | Memory Usage (MB) | |
| |---|---|---|---| |
| | J6M | — | — | — | |
| |
| --- |
| |
| ## Model Overview |
| |
| ### Core Design |
| |
| HENet-tiny extracts multi-view camera features; a single-frame `SingleBevFormerViewTransformer` (with `LearnedPositionalEncoding`, `HorizonSpatialCrossAttention` and `HorizonMultiScaleDeformableAttention`) lifts them to a `128×128` BEV grid covering `[-51.2m, 51.2m]²`. The BEV feature drives five task heads: dynamic/static 3D detection (`DenseBEVSparseHead`, SparseBEV-style sparse queries with `MemoryBank` temporal tracking), occupancy (`BEVOCCHead2D`, 18 classes), discrete map (`BEVFormerDet2DDecoder`, stop line), and online vectorized map (`MaptrDetDecoder`, 3 classes). |
| |
| - **Task type**: BEV multitask perception (dynamic det + static det + occupancy + discrete map + online map). |
| - **backbone**: HENet-tiny (`type=HENet`, `block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`, `GroupDWCB`/`AltDWCB`/`DWCB` blocks, `S2DDown` downsampling), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint. |
| - **neck**: `FPN` (`in_strides=[32]` → `out_strides=[32]`, `_dim_=256`) + `BevFuseModule` (`input_c=384+256` → `fuse_c=256`; lidar branch disabled, `lidar_input=False`). |
| - **Dynamic det head**: `DenseBEVSparseHead` (8 classes, `num_anchor=384`, `num_decoder=6`, `MemoryBank` with `num_temp_instances=128` for tracking). |
| - **Static det head**: `DenseBEVSparseHead` (2 classes: `traffic_cone`, `barrier`; `num_anchor=200`). |
| - **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 18 classes). |
| - **Discrete map head**: `BEVFormerDet2DDecoder` (1 class `stop_line`, `num_query=100`). |
| - **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (3 classes: `divider`, `ped_crossing`, `boundary`; 20 points per polyline). |
| - **Loss functions**: det — `FocalLoss` (cls) + `L1Loss` (reg) + `CrossEntropyLoss` (cns) + `GaussianFocalLoss` (yns); occ — `CrossEntropyLoss`; maptr — `MapTRCriterion`; discrete map — `BevFormer2DCriterion`. |
| - **Model input**: 6-camera multi-view images `(B,6,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only, no lidar at inference). |
| - **Model output**: dynamic/static 3D boxes + occ voxel semantics + stop-line boxes + map polylines. |
|
|
| ### Official Repo and Paper |
|
|
| SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV |
| Paper: https://arxiv.org/abs/2308.09244 |
| MapTR repo: https://github.com/hustvl/MapTR |
| Paper: https://arxiv.org/abs/2208.14437 |
|
|
| Note: The camera backbone HENet and the multitask assembly are HEAL in-house implementations; the official repos use different backbones and single-task setups. |
|
|
