File size: 4,245 Bytes
ebe01a0 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: other
tags:
- heal
- horizon
- bev
---
# BEV Multitask + HENet-tiny
`bev_multitask_henet_tinym` is a camera-only BEV multitask perception model: HENet-tiny backbone + single-frame BevFormer view transformer, jointly performing five tasks — dynamic object 3D detection (with tracking), static object 3D detection, occupancy prediction, discrete map (stop line) detection, and online vectorized map (MapTR). It covers the full pipeline from 3-stage float training → quantization calibration → HBIR export → HBM compilation → inference.
---
## Deployment Metrics
### Model Parameters
| Model | Model Input | Backbone | Neck | Model Output |
|---|---|---|---|---|
| BEVFusionMultitask | 6-camera multi-view images `(B,6,3,512,960)` | HENet-tiny | FPN + SingleBevFormerViewTransformer | dynamic/static 3D boxes `(B,N,cls+reg)`; occ voxel semantics; discrete-map 2D boxes; map polylines |
### Accuracy Metrics
| March | Metric | float | calibration | qat | hbm |
| --- | --- | --- | --- | --- | --- |
| J6M | — | — | — | — | — |
> Accuracy numbers not yet measured; fill in after evaluation. Results use `march = March.NASH_M` (J6M) configuration; this task has no QAT stage (qat column is `—`).
>
> HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.1 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2).
### Performance Metrics
> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.
> Data source: `tools/test_perf_all.py` (default sampling interval `-d 200ms`, perf-time 1).
| March | latency (ms) | fps | Memory Usage (MB) |
|---|---|---|---|
| J6M | — | — | — |
---
## Model Overview
### Core Design
HENet-tiny extracts multi-view camera features; a single-frame `SingleBevFormerViewTransformer` (with `LearnedPositionalEncoding`, `HorizonSpatialCrossAttention` and `HorizonMultiScaleDeformableAttention`) lifts them to a `128×128` BEV grid covering `[-51.2m, 51.2m]²`. The BEV feature drives five task heads: dynamic/static 3D detection (`DenseBEVSparseHead`, SparseBEV-style sparse queries with `MemoryBank` temporal tracking), occupancy (`BEVOCCHead2D`, 18 classes), discrete map (`BEVFormerDet2DDecoder`, stop line), and online vectorized map (`MaptrDetDecoder`, 3 classes).
- **Task type**: BEV multitask perception (dynamic det + static det + occupancy + discrete map + online map).
- **backbone**: HENet-tiny (`type=HENet`, `block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`, `GroupDWCB`/`AltDWCB`/`DWCB` blocks, `S2DDown` downsampling), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint.
- **neck**: `FPN` (`in_strides=[32]` → `out_strides=[32]`, `_dim_=256`) + `BevFuseModule` (`input_c=384+256` → `fuse_c=256`; lidar branch disabled, `lidar_input=False`).
- **Dynamic det head**: `DenseBEVSparseHead` (8 classes, `num_anchor=384`, `num_decoder=6`, `MemoryBank` with `num_temp_instances=128` for tracking).
- **Static det head**: `DenseBEVSparseHead` (2 classes: `traffic_cone`, `barrier`; `num_anchor=200`).
- **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 18 classes).
- **Discrete map head**: `BEVFormerDet2DDecoder` (1 class `stop_line`, `num_query=100`).
- **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (3 classes: `divider`, `ped_crossing`, `boundary`; 20 points per polyline).
- **Loss functions**: det — `FocalLoss` (cls) + `L1Loss` (reg) + `CrossEntropyLoss` (cns) + `GaussianFocalLoss` (yns); occ — `CrossEntropyLoss`; maptr — `MapTRCriterion`; discrete map — `BevFormer2DCriterion`.
- **Model input**: 6-camera multi-view images `(B,6,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only, no lidar at inference).
- **Model output**: dynamic/static 3D boxes + occ voxel semantics + stop-line boxes + map polylines.
### Official Repo and Paper
SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV
Paper: https://arxiv.org/abs/2308.09244
MapTR repo: https://github.com/hustvl/MapTR
Paper: https://arxiv.org/abs/2208.14437
Note: The camera backbone HENet and the multitask assembly are HEAL in-house implementations; the official repos use different backbones and single-task setups.
|