--- license: apache-2.0 pipeline_tag: object-detection tags: - autoware - ros2 - autonomous-driving - camera - multi-view - 3d-object-detection - streampetr - tensorrt - onnx --- # StreamPETR for Autoware (`camera_streampetr`) Camera-only multi-view 3D object detection models, used by the [`autoware_camera_streampetr`](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_camera_streampetr) node in [Autoware](https://github.com/autowarefoundation/autoware). The models follow the **StreamPETR** [1] architecture, an object-centric temporal modeling approach for efficient multi-view 3D detection, and run with TensorRT inside Autoware. This is the first camera-only 3D object detection node in Autoware: it detects 3D objects from surround-view camera images without any LiDAR input. The models are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from the ONNX files on first launch. ## Model overview | | | | --- | --- | | Task | 3D object detection (oriented bounding boxes) from multi-view camera images | | Architecture | StreamPETR (image backbone + 3D position embedding + temporal propagation head) | | Detected classes | `CAR`, `TRUCK`, `BUS`, `BICYCLE`, `PEDESTRIAN` | | Cameras | 5 views: `CAM_FRONT`, `CAM_FRONT_LEFT`, `CAM_BACK_LEFT`, `CAM_FRONT_RIGHT`, `CAM_BACK_RIGHT` | | Input resolution | 640 x 480 per camera (after node-side resize/crop/normalize) | | Detection range | `[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]` m from `base_link` | | Temporal modeling | Propagated object memory (input length 1024, output length 1280), 5400 proposals | | Runtime | TensorRT (FP16 by default, FP32 selectable) via the `autoware_camera_streampetr` ROS 2 node | | Format | ONNX (Autoware builds the TensorRT engines locally on first launch) | | License | Apache-2.0 | The network is split into three ONNX sub-models, matching how the node consumes them: 1. **Image feature extractor (backbone)**: `simplify_extract_img_feat.onnx` 2. **Position embedding**: `simplify_position_embedding.onnx` 3. **Detection head with temporal memory**: `simplify_pts_head_memory.onnx` Pre-processing (undistortion if needed, resize, crop, normalization, ego-vehicle masking) and post-processing (circle NMS, IoU NMS, per-class confidence thresholding, yaw normalization) run in the node, not in the ONNX graphs. ## Files | File | Description | | --- | --- | | `simplify_extract_img_feat.onnx` | Image feature extractor (backbone) | | `simplify_position_embedding.onnx` | 3D position embedding network | | `simplify_pts_head_memory.onnx` | Detection head with temporal memory | | `ml_package_camera_streampetr.param.yaml` | Model parameters (classes, detection range, input resolution, camera count, proposal and memory lengths) | | `deploy_metadata.yaml` | Deployment metadata recording the artifact version of this repository | > **TensorRT engines are not distributed here.** TensorRT engines are specific to the GPU architecture and > TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files > on first launch (or via `build_only:=true`). ## Inputs and outputs (as used by the node) **Inputs** - `~/input/camera*/image` (`sensor_msgs/msg/Image` or `sensor_msgs/msg/CompressedImage`): camera images, compressed or uncompressed. - `~/input/camera*/camera_info` (`sensor_msgs/msg/CameraInfo`): camera intrinsics. The node is optimized for multi-camera systems where camera topics arrive sequentially rather than all at once: each image is preprocessed and stored on the GPU as it arrives, and the forward pass is triggered when the anchor camera's image arrives and all images are synchronized within `max_camera_time_diff`. **Output** - `~/output/objects` (`autoware_perception_msgs/msg/DetectedObjects`): oriented 3D boxes with class and score. - `latency/*` (`autoware_internal_debug_msgs/msg/Float64Stamped`): per-stage timing topics for debugging. On an RTX 3090 the package reports a total processing time of about 26 ms per inference (the reported time between consecutive predictions is about 111 ms). ## Usage in Autoware The node expects these artifacts in `~/autoware_data/ml_models/camera_streampetr/` (the default `model_path`) and launches with: ```bash ros2 launch autoware_camera_streampetr streampetr.launch.xml \ model_path:=$HOME/autoware_data/ml_models/camera_streampetr ``` Add `build_only:=true` to build the TensorRT engines from the ONNX files as a one-off pre-task. The `ml_package_camera_streampetr.param.yaml` shipped here is loaded by the launch file and carries the model-dependent parameters; node behavior parameters live in the package's own `config/camera_streampetr.param.yaml`. See the [package README](https://github.com/autowarefoundation/autoware_universe/tree/main/perception/autoware_camera_streampetr) for the full parameter reference. ## Training The model was trained and exported with the AWML StreamPETR project: - Training datasets, schedules, and evaluation metrics for these specific weights are not publicly documented. ## Limitations - The model only predicts objects inside the detection range above; the node filters out anything beyond it. - Only the five classes above are detected. Other road users fall outside the label set. - The node assumes undistorted input images (or enables undistortion itself), accurate camera calibration, cameras synchronized within `max_camera_time_diff`, and tf transforms between the camera frames, `base_link`, and `map` (for ego motion compensation). - Predictions do not include object velocity. ## Provenance | | | | --- | --- | | Original source | `https://awf.ml.dev.web.auto/perception/models/streampetr/v1/` | | This repository | `AutowareFoundation/camera_streampetr`, tag `v1.0` | The `v1.0` tag corresponds to the `streampetr/v1` artifact version at the original hosting path. ## Citation ```bibtex @article{wang2023streampetr, title = {Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection}, author = {Wang, Shihao and Liu, Yingfei and Wang, Tiancai and Li, Ying and Zhang, Xiangyu}, journal = {arXiv preprint arXiv:2303.11926}, year = {2023} } ``` ## References - [1] Wang et al., "Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection", arXiv:2303.11926, 2023. - AWML StreamPETR project: - Autoware: