File size: 4,611 Bytes
64009a6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 | ---
license: other
tags:
- heal
- horizon
---
# E2E Diffusion Planner + HENet-tiny
`e2e_planner` is an end-to-end autonomous-driving model on NAVSIM: an 8-camera BEV multitask perception front-end (HENet-tiny + single-frame BevFormer view transformer; dynamic/static 3D detection, occupancy, discrete map, online MapTR) feeding a BEV-feature-conditioned Diffusion Planner that outputs future ego trajectories. It covers the full pipeline from 5-stage float training → quantization calibration → HBIR export → HBM compilation → NAVSIM closed-loop metric evaluation.
---
## Deployment Metrics
### Model Parameters
| Model | Model Input | Backbone | Neck | Model Output |
|---|---|---|---|---|
| E2EPlanner (BEVFusionMultitask + Diffusion_Planner) | 8-camera multi-view images `(B,8,3,512,960)` + ego state + route lanes | HENet-tiny | FPN + SingleBevFormerViewTransformer | future ego trajectory `(1,8,4)` via diffusion |
### Accuracy Metrics
| March | Metric | float | calibration | hbm |
| --- | --- | --- | --- | --- |
| J6M | PDMS | 0.8223 | 0.8142 | — |
> Measured 2026-10-17 on NAVSIM navtest with `HAT_USE_NEIGHBORS_FUTURE=0` and `dequantize_bev_embed=False`. No QAT stage. HBM accuracy not yet measured.
>
> HEAL version: heal 0.6.0 / hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5 (board UCP 3.16.2).
### Performance Metrics
> **Performance benchmark**: FPS is measured with single-core 8 threads; latency is single-core single-thread; memory is peak DDR usage.
> Data source: `tools/perf.py` (`--perf-time 1`; DDR sampled by `hrt_ucp_monitor -d 200`). Measured 2026-10-19 (`dequantize_bev_embed=False`). Toolchain: hbdk4-compiler 4.13.3 / horizon_plugin_pytorch 3.4.5. No `native::` CPU ops.
| March | latency (ms) | fps | Memory Usage (MB) |
|---|---|---|---|
| J6M | 99.14 (BPU 99.07) | 10.11 | 306.30 |
| J6P | 37.84 (BPU 37.78) | 88.23 | 242.20 |
---
## Model Overview
### Core Design
The model has two halves joined at the BEV feature: an 8-camera multitask perception front-end producing a `128×192` BEV grid, and a BEV-conditioned Diffusion Planner. HENet-tiny extracts multi-view camera features; a single-frame `SingleBevFormerViewTransformer` lifts them to BEV; five task heads (`DenseBEVSparseHead` dynamic/static, `BEVOCCHead2D` occupancy, `BEVFormerDet2DDecoder` discrete map, `MaptrDetDecoder` online map) share the BEV feature. The planner encodes the pooled BEV feature into 384 tokens, conditions a 3-layer DiT via a route-lane `RouteEncoder`, and denoises future ego waypoints over 5 reverse diffusion steps.
- **Task type**: End-to-end planning (perception + planning joint).
- **Perception backbone**: HENet-tiny (`block_nums=[4,3,8,6]`, `embed_dims=[64,128,192,384]`), initialized from the ImageNet-pretrained `henet_tinym` float checkpoint.
- **Perception neck**: `FPN` + `SingleBevFormerViewTransformer` (`128×192` BEV, `[-51.2m, 102.4m] × [-51.2m, 51.2m]`).
- **Dynamic det head**: `DenseBEVSparseHead` (3 classes; `num_anchor=384`, `MemoryBank` 128 temporal instances).
- **Static det head**: `DenseBEVSparseHead` (4 classes; `num_anchor=200`).
- **Occupancy head**: `BevformerOccDetDecoder` + `BEVOCCHead2D` (`Dz=16`, 9 classes).
- **Discrete map head**: `BEVFormerDet2DDecoder` (2 classes: stop_line/crosswalk).
- **Online map head**: `MaptrDetDecoder` + `MapInstanceDetectorHead` (1 class `lanes`, SD-map fusion, 20 points per polyline).
- **Planner encoder**: `Diffusion_Planner_Encoder` (`BEVFusionEncoder` ×8 downsample → 384 tokens, 3-layer `FusionEncoder` transformer).
- **Planner decoder**: 3-layer `DiT` (`hidden_dim=256`, 8 heads, `RouteEncoder`), `model_type="x_start"`, `diffusion_pred_steps=5`, linear VP noise schedule.
- **Loss functions**: perception — `FocalLoss` + `L1Loss` + `CrossEntropyLoss` + `GaussianFocalLoss` / `CrossEntropyLoss` / `MapTRCriterion` / `BevFormer2DCriterion` per head; planner — `DiffusionLoss` (ego 2.0 + collision 1.0).
- **Model input**: 8-camera multi-view images `(B,8,3,512,960)` (YUV444 + `Normalize(128,128)`; camera-only) + ego state + route lanes.
- **Model output**: 8-step future ego trajectory `(1,8,4)`.
### Official Repo and Paper
NAVSIM benchmark: https://github.com/autonomousvision/navsim
Paper: https://arxiv.org/abs/2406.15349
SparseBEV-style det head repo: https://github.com/MCG-NJU/SparseBEV
Paper: https://arxiv.org/abs/2308.09244
MapTR repo: https://github.com/hustvl/MapTR
Paper: https://arxiv.org/abs/2208.14437
Note: The camera backbone HENet and the E2E assembly are HEAL in-house implementations; the official repos use different backbones and pipelines.
|