| --- |
| license: other |
| tags: |
| - heal |
| - horizon |
| --- |
| |
| # E2E Diffusion Planner + HENet-tiny |
|
|
| `e2e_planner` is an end-to-end autonomous-driving model on NAVSIM: an 8-camera BEV multitask perception front-end (HENet-tiny + single-frame BevFormer view transformer; dynamic/static 3D detection, occupancy, discrete map, online MapTR) feeding a BEV-feature-conditioned Diffusion Planner that outputs future ego trajectories. It covers the full pipeline from 5-stage float training → quantization calibration → HBIR export → HBM compilation → NAVSIM closed-loop metric evaluation. |
|
|
| --- |
|
|
| ## Deployment Metrics |
|
|
| ### Model Parameters |
|
|
| | Model | Model Input | Backbone | Neck | Model Output | |
| |---|---|---|---|---| |
| | E2EPlanner (BEVFusionMultitask + Diffusion_Planner) | 8-camera multi-view images `(B,8,3,512,960)` + ego state + route lanes | HENet-tiny | FPN + SingleBevFormerViewTransformer | future ego trajectory `(1,8,4)` via diffusion | |
| |
| ### Accuracy Metrics |
| |
| | March | Metric | float | calibration | hbm | |
| | --- | --- | --- | --- | --- | |
| | J6M | PDMS | 0.8223 | 0.8142 | — | |
| |
| > Measured 2026-10-17 on NAVSIM navtest with `HAT_USE_NEIGHBORS_FUTURE=0` and `dequantize_bev_embed=False`. No QAT stage. HBM accuracy not yet measured. |
| > |
| > HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2). |
|
|
| ### Performance Metrics |
|
|
| > **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage. |
| > Data source: `tools/perf.py` (`--perf-time 1`; DDR sampled by `hrt_ucp_monitor -d 200`). Measured 2026-10-19 (`dequantize_bev_embed=False`). Toolchain: hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5. No `native::` CPU ops. |
|
|
| | March | latency (ms) | fps | Memory Usage (MB) | |
| |---|---|---|---| |
| | J6M | 99.14 (BPU 99.07) | 10.11 | 306.30 | |
| | J6P | 37.84 (BPU 37.78) | 88.23 | 242.20 | |
|
|
| --- |
|
|
| ## Model Overview |
|
|
| ### Core Design |
|
|
| The model has two halves joined at the BEV feature: an 8-camera multitask perception front-end producing a `128×192` BEV grid, and a BEV-conditioned Diffusion Planner. HENet-tiny extracts multi-view camera features; a single-frame `SingleBevFormerViewTransformer` lifts them to BEV; five task heads (`DenseBEVSparseHead` dynamic/static, `BEVOCCHead2D` occupancy, `BEVFormerDet2DDecoder` discrete map, `MaptrDetDecoder` online map) share the BEV feature. The planner encodes the pooled BEV feature into 384 tokens, conditions a 3-layer DiT via a route-lane `RouteEncoder`, and denoises future ego waypoints over 5 reverse diffusion steps. |
|
|
| - **Task type**: End-to-end planning (perception + planning joint). |
| - **Perception backbone**: HENet-tiny (`block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint. |
| - **Perception neck**: `FPN` + `SingleBevFormerViewTransformer` (`128×192` BEV, `[-51.2m, 102.4m] × [-51.2m, 51.2m]`). |
| - **Dynamic det head**: `DenseBEVSparseHead` (3 classes; `num_anchor=384`, `MemoryBank` 128 temporal instances). |
| - **Static det head**: `DenseBEVSparseHead` (4 classes; `num_anchor=200`). |
| - **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 9 classes). |
| - **Discrete map head**: `BEVFormerDet2DDecoder` (2 classes: stop_line/crosswalk). |
| - **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (1 class `lanes`, SD-map fusion, 20 points per polyline). |
| - **Planner encoder**: `Diffusion_Planner_Encoder` (`BEVFusionEncoder` ×8 downsample → 384 tokens, 3-layer `FusionEncoder` transformer). |
| - **Planner decoder**: 3-layer `DiT` (`hidden_dim=256`, 8 heads, `RouteEncoder`), `model_type="x_start"`, `diffusion_pred_steps=5`, linear VP noise schedule. |
| - **Loss functions**: perception — `FocalLoss` + `L1Loss` + `CrossEntropyLoss` + `GaussianFocalLoss` / `CrossEntropyLoss` / `MapTRCriterion` / `BevFormer2DCriterion` per head; planner — `DiffusionLoss` (ego 2.0 + collision 1.0). |
| - **Model input**: 8-camera multi-view images `(B,8,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only) + ego state + route lanes. |
| - **Model output**: 8-step future ego trajectory `(1,8,4)`. |
|
|
| ### Official Repo and Paper |
|
|
| NAVSIM benchmark: https://github.com/autonomousvision/navsim |
| Paper: https://arxiv.org/abs/2406.15349 |
| SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV |
| Paper: https://arxiv.org/abs/2308.09244 |
| MapTR repo: https://github.com/hustvl/MapTR |
| Paper: https://arxiv.org/abs/2208.14437 |
|
|
| Note: The camera backbone HENet and the E2E assembly are HEAL in-house implementations; the official repos use different backbones and pipelines. |
|
|
