--- license: apache-2.0 pipeline_tag: image-segmentation tags: - autoware - ros2 - autonomous-driving - lidar - point-cloud - semantic-segmentation - frnet - tensorrt - onnx --- # FRNet for Autoware (`lidar_frnet`) LiDAR semantic segmentation models used by the [`autoware_lidar_frnet`](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_frnet) node in [Autoware](https://github.com/autowarefoundation/autoware). The models follow the **FRNet** (Frustum-Range Network) [1] architecture and perform per-point 3D semantic segmentation on LiDAR data (x, y, z, intensity). They are exported as ONNX and run with TensorRT inside Autoware; the node builds the TensorRT engine from the ONNX file on first launch. ## Model overview | | | | --- | --- | | Task | 3D semantic segmentation of LiDAR point clouds (per-point class ID and probability) | | Architecture | FRNet (Frustum-Range Network) | | Classes | 27 semantic classes (see list below) | | Runtime | TensorRT (FP16 by default) via the `autoware_lidar_frnet` ROS 2 node | | Format | ONNX (Autoware builds the TensorRT engine locally on first launch) | | License | Apache-2.0 | Segmented classes (as defined in the `ml_package_frnet_*.param.yaml` files): `drivable_surface`, `other_flat_surface`, `sidewalk`, `manmade`, `vegetation`, `car`, `bus`, `emergency_vehicle`, `train`, `truck`, `tractor_unit`, `semi_trailer`, `construction_vehicle`, `forklift`, `kart`, `motorcycle`, `bicycle`, `pedestrian`, `personal_mobility`, `animal`, `pushable_pullable`, `traffic_cone`, `debris`, `stroller`, `other_stuff`, `noise+ghost_point`, `unknown` Pre-processing (range-image projection, frustum construction) and post-processing (per-point argmax, probability-based filtering, visualization coloring) run in the node, not in the ONNX graph. ## Variants in this repository FRNet uses range images, so each variant is tied to the field of view and resolution of a specific LiDAR sensor: | Variant | Sensor | FOV up / down [deg] | Frustum (W x H) | Interpolation (W x H) | Points (min / opt / max) | | --- | --- | --- | --- | --- | --- | | `frnet_ot128` | HESAI OT128 | `15.0` / `-25.0` | `1024 x 128` | `4096 x 128` | `5000 / 80000 / 160000` | | `frnet_qt128` | HESAI QT128 | `52.6` / `-52.6` | `256 x 128` | `1024 x 128` | `5000 / 60000 / 120000` | Both variants share the same 27-class label set and color palette. The point count ranges define the dynamic TensorRT optimization profile for the input point cloud. ## Files | File | Description | | --- | --- | | `frnet_ot128.onnx` | FRNet network, HESAI OT128 variant | | `frnet_qt128.onnx` | FRNet network, HESAI QT128 variant | | `ml_package_frnet_ot128.param.yaml` | Model parameters for `frnet_ot128` (FOV, frustum/interpolation size, classes, palette, profile ranges) | | `ml_package_frnet_qt128.param.yaml` | Model parameters for `frnet_qt128` (FOV, frustum/interpolation size, classes, palette, profile ranges) | | `deploy_metadata.yaml` | Deployment metadata recording the artifact version of this repository | > **TensorRT engines are not distributed here.** TensorRT engines are specific to the GPU architecture and > TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files > on first launch (or via `build_only:=true`). ## Inputs and outputs (as used by the node) **Input**: `~/input/pointcloud` (`sensor_msgs/msg/PointCloud2`). The node operates on the x, y, z and intensity fields and auto-detects the input layout on the first received message; supported layouts are `XYZIRCAEDT`, `XYZIRADRT`, `XYZIRC` and `XYZI`. **Outputs**: | Topic | Type | Description | | --- | --- | --- | | `~/output/pointcloud/segmentation` | `sensor_msgs/msg/PointCloud2` | XYZ cloud with class ID and probability fields | | `~/output/pointcloud/visualization` | `sensor_msgs/msg/PointCloud2` | XYZ cloud with RGB field (palette colors) | | `~/output/pointcloud/filtered` | `sensor_msgs/msg/PointCloud2` | Cloud with the configured filter classes removed (default: `drivable_surface`), in the requested `filter.output_format` | The node also publishes processing time and latency debug topics and `/diagnostics`. ## Usage in Autoware Autoware's setup tooling (the Ansible `artifacts` role) downloads these artifacts to `~/autoware_data/ml_models/lidar_frnet/`. The node then loads `frnet_.onnx` together with the matching `ml_package_frnet_.param.yaml` from `model_path`: ```bash ros2 launch autoware_lidar_frnet lidar_frnet.launch.xml \ sensor_model:=ot128 \ model_path:=$HOME/autoware_data/ml_models/lidar_frnet ``` `sensor_model` accepts `ot128` (HESAI OT128) or `qt128` (HESAI QT128). Add `build_only:=true` to build the TensorRT engine from the ONNX as a one-off pre-task. See the [package README](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_frnet) for the full parameter reference. ## Training The models were trained with AWML () on the T4Dataset, using approximately 16,000 frames (4,000 frames x 4 surrounding sensors), as documented in the package README. - FRNet project page: - FRNet paper: arXiv:2312.04484 ## Provenance and versioning | HF tag | Original artifact source | | --- | --- | | `v2.0` | `https://awf.ml.dev.web.auto/perception/models/frnet/v2/` | The model and parameter files (`frnet_*.onnx`, `ml_package_frnet_*.param.yaml`) byte-match the artifacts previously hosted at the source path above; `deploy_metadata.yaml` is generated for this repository and has no upstream counterpart. Consumers should pin a tag (`--revision v2.0`), never `main`. ## Limitations - FRNet uses range images, so the input point cloud must be referenced to the sensor's origin. - Each variant is tuned to its sensor's FOV and resolution; differences in your sensor's specifications (FOV, horizontal/vertical resolution) may affect performance. - Only the two sensor variants above are provided; other LiDAR models require retraining or fine-tuning. ## Citation ```bibtex @article{xu2025frnet, title = {FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation}, author = {Xu, Xiang and Kong, Lingdong and Shuai, Hui and Liu, Qingshan}, journal = {IEEE Transactions on Image Processing}, volume = {34}, pages = {2173--2186}, year = {2025}, doi = {10.1109/TIP.2025.3550011} } ``` ## References - [1] X. Xu, L. Kong, H. Shuai and Q. Liu, "FRNet: Frustum-Range Networks for Scalable LiDAR Segmentation", IEEE Transactions on Image Processing, vol. 34, pp. 2173-2186, 2025. arXiv:2312.04484. - [2] AWML: - [3] FRNet project page: