feat: add lidar_apollo_instance_segmentation v1.0 artifacts (from awf.ml.dev.web.auto/perception/models/lidar_apollo_instance_segmentation)

0b4e73b verified 1 day ago

preview code

Raw

History Blame Contribute Delete

7.32 kB

metadata

license: apache-2.0
pipeline_tag: object-detection
tags:
  - autoware
  - ros2
  - autonomous-driving
  - lidar
  - point-cloud
  - instance-segmentation
  - apollo
  - tensorrt
  - onnx

Apollo LiDAR Instance Segmentation for Autoware (`lidar_apollo_instance_segmentation`)

CNN-based LiDAR instance segmentation models, used by the autoware_lidar_apollo_instance_segmentation node in Autoware.

The models originate from the Baidu Apollo 3D obstacle perception stack (CNN segmentation). The original Apollo caffemodels were converted to ONNX for use in Autoware, and the node builds a TensorRT engine from the ONNX file on first launch. The node rasterizes the input point cloud into a bird's-eye-view feature map, runs the network, and clusters the per-cell predictions into labeled obstacle instances.

Model overview


Task	Instance segmentation of LiDAR point clouds into labeled obstacle clusters
Architecture	Apollo CNN segmentation (CNNSeg) on a bird's-eye-view feature map, plus obstacle clustering in the node
Detected classes	Apollo meta types mapped to Autoware labels: `CAR` (small vehicle), `BUS` (big vehicle), `MOTORCYCLE` (non-motorized vehicle), `PEDESTRIAN`; unmatched clusters are `UNKNOWN`
Runtime	TensorRT (FP32 by default) via the `autoware_lidar_apollo_instance_segmentation` ROS 2 node
Format	ONNX, converted from Apollo caffemodels (Autoware builds the TensorRT engine locally on first launch)
License	Apache-2.0 (weights derived from Apollo, which is Apache-2.0 licensed)

Variants in this repository

One ONNX model per supported LiDAR sensor, selected via the model launch argument:

Variant	Launch value	Feature map grid	Range [m]	Intensity feature
`vlp-16` (Velodyne VLP-16)	`model_16`	672 x 672	70	yes
`hdl-64` (Velodyne HDL-64)	`model_64`	672 x 672	70	yes
`vls-128` (Velodyne VLS-128)	`model_128` (default)	864 x 864	90	no

The per-variant parameter files (vlp-16.param.yaml, hdl-64.param.yaml, vls-128.param.yaml) live in the consuming package under config/, not in this repository. The values above come from those files. Supported LiDARs are the Velodyne 16, 64, and 128 beam sensors, but the package README notes that other LiDARs such as the Velodyne 32 can also be used with good accuracy.

Files

File	Description
`vlp-16.onnx`	CNN segmentation model for Velodyne VLP-16
`hdl-64.onnx`	CNN segmentation model for Velodyne HDL-64
`vls-128.onnx`	CNN segmentation model for Velodyne VLS-128
`deploy_metadata.yaml`	Deployment metadata recording the artifact version of this repository

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Input: input/pointcloud (sensor_msgs/msg/PointCloud2), point cloud data from LiDAR sensors (default remap: /sensing/lidar/pointcloud).

Outputs:

output/labeled_clusters (tier4_perception_msgs/msg/DetectedObjectsWithFeature): detected objects with labeled point cloud clusters.
debug/instance_pointcloud (sensor_msgs/msg/PointCloud2): segmented point cloud for visualization.

Pre-processing (bird's-eye-view feature map generation) and post-processing (2D clustering of the network output into obstacle instances, score thresholding via score_threshold) run in the node, not in the ONNX graphs.

Usage in Autoware

The node expects these artifacts under ~/autoware_data/ml_models/lidar_apollo_instance_segmentation/ and launches with, e.g.:

ros2 launch autoware_lidar_apollo_instance_segmentation lidar_apollo_instance_segmentation.launch.xml \
  model:=model_128 \
  data_path:=$HOME/autoware_data/ml_models

model:=model_16, model_64, or model_128 selects vlp-16.onnx, hdl-64.onnx, or vls-128.onnx respectively, together with the matching parameter file from the package's config/ directory. Add build_only:=true to build the TensorRT engine from the ONNX as a one-off pre-task. See the package README for the full parameter reference.

Training

There is no training code for these models. The weights were trained by the Baidu Apollo project and released as caffemodels, which were later converted to ONNX for Autoware. Training datasets, schedules, and metrics are not publicly documented.

Origin and related projects:

Apollo autonomous driving platform (original caffemodels and CNN segmentation design): https://github.com/ApolloAuto/apollo
Initial Autoware integration of the Apollo CNN segmentation detector: https://github.com/k0suke-murakami/autoware_perception
TensorRT wrapper library historically used by the node (current code uses autoware_tensorrt_common): https://github.com/lewes6369/tensorRTWrapper

The original caffemodel sources (pinned Apollo commits) are listed in the package README:

VLP-16: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne16/deploy.caffemodel (Apollo commit 88bfa5a)
HDL-64: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne64/deploy.caffemodel (Apollo commit 88bfa5a)
VLS-128: modules/perception/production/data/perception/lidar/models/cnnseg/velodyne128/deploy.caffemodel (Apollo commit 91844c8)

Provenance


Original hosting	`https://awf.ml.dev.web.auto/perception/models/lidar_apollo_instance_segmentation/` (unversioned)
This repository	`AutowareFoundation/lidar_apollo_instance_segmentation`, tag `v1.0`

The v1.0 tag corresponds to the exact file set previously served from the unversioned awf.ml.dev.web.auto path.

Limitations

Each ONNX file is tuned for a specific Velodyne sensor (16, 64, or 128 beams); other LiDARs may work with good accuracy but are not the intended configuration.
Only the classes listed above are produced; other road users are reported as UNKNOWN.
No training code or training data documentation exists, so the models cannot be retrained or fine-tuned from public sources.

References

Apollo 3D Obstacle Perception description: https://github.com/ApolloAuto/apollo/blob/r7.0.0/docs/specs/3d_obstacle_perception.md
Consuming Autoware package: https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_lidar_apollo_instance_segmentation

Legal notice

The model weights are derived from the Baidu Apollo project, which is licensed under the Apache License 2.0. The consuming Autoware package additionally incorporates code from the Apollo project (Apache-2.0), the tensorRTWrapper library (MIT), and the autoware_perception integration (Apache-2.0); see the package README for the full license texts.

Apollo LiDAR Instance Segmentation for Autoware (lidar_apollo_instance_segmentation)