StreamPETR for Autoware (camera_streampetr)

Camera-only multi-view 3D object detection models, used by the autoware_camera_streampetr node in Autoware.

The models follow the StreamPETR [1] architecture, an object-centric temporal modeling approach for efficient multi-view 3D detection, and run with TensorRT inside Autoware. This is the first camera-only 3D object detection node in Autoware: it detects 3D objects from surround-view camera images without any LiDAR input. The models are exported as ONNX so they can be deployed across hardware; Autoware builds the TensorRT engines from the ONNX files on first launch.

Model overview

Task	3D object detection (oriented bounding boxes) from multi-view camera images
Architecture	StreamPETR (image backbone + 3D position embedding + temporal propagation head)
Detected classes	CAR, TRUCK, BUS, BICYCLE, PEDESTRIAN
Cameras	5 views: CAM_FRONT, CAM_FRONT_LEFT, CAM_BACK_LEFT, CAM_FRONT_RIGHT, CAM_BACK_RIGHT
Input resolution	640 x 480 per camera (after node-side resize/crop/normalize)
Detection range	[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0] m from base_link
Temporal modeling	Propagated object memory (input length 1024, output length 1280), 5400 proposals
Runtime	TensorRT (FP16 by default, FP32 selectable) via the autoware_camera_streampetr ROS 2 node
Format	ONNX (Autoware builds the TensorRT engines locally on first launch)
License	Apache-2.0

The network is split into three ONNX sub-models, matching how the node consumes them:

1. Image feature extractor (backbone): simplify_extract_img_feat.onnx
2. Position embedding: simplify_position_embedding.onnx
3. Detection head with temporal memory: simplify_pts_head_memory.onnx

Pre-processing (undistortion if needed, resize, crop, normalization, ego-vehicle masking) and post-processing (circle NMS, IoU NMS, per-class confidence thresholding, yaw normalization) run in the node, not in the ONNX graphs.

Files

File	Description
simplify_extract_img_feat.onnx	Image feature extractor (backbone)
simplify_position_embedding.onnx	3D position embedding network
simplify_pts_head_memory.onnx	Detection head with temporal memory
ml_package_camera_streampetr.param.yaml	Model parameters (classes, detection range, input resolution, camera count, proposal and memory lengths)
deploy_metadata.yaml	Deployment metadata recording the artifact version of this repository

TensorRT engines are not distributed here. TensorRT engines are specific to the GPU architecture and TensorRT version they are built on and are not portable, so Autoware builds them locally from the ONNX files on first launch (or via build_only:=true).

Inputs and outputs (as used by the node)

Inputs

• ~/input/camera*/image (sensor_msgs/msg/Image or sensor_msgs/msg/CompressedImage): camera images, compressed or uncompressed.
• ~/input/camera*/camera_info (sensor_msgs/msg/CameraInfo): camera intrinsics.

The node is optimized for multi-camera systems where camera topics arrive sequentially rather than all at once: each image is preprocessed and stored on the GPU as it arrives, and the forward pass is triggered when the anchor camera's image arrives and all images are synchronized within max_camera_time_diff.

Output

• ~/output/objects (autoware_perception_msgs/msg/DetectedObjects): oriented 3D boxes with class and score.
• latency/* (autoware_internal_debug_msgs/msg/Float64Stamped): per-stage timing topics for debugging. On an RTX 3090 the package reports a total processing time of about 26 ms per inference (the reported time between consecutive predictions is about 111 ms).

Usage in Autoware

The node expects these artifacts in ~/autoware_data/ml_models/camera_streampetr/ (the default model_path) and launches with:

ros2 launch autoware_camera_streampetr streampetr.launch.xml \
  model_path:=$HOME/autoware_data/ml_models/camera_streampetr

Add build_only:=true to build the TensorRT engines from the ONNX files as a one-off pre-task. The ml_package_camera_streampetr.param.yaml shipped here is loaded by the launch file and carries the model-dependent parameters; node behavior parameters live in the package's own config/camera_streampetr.param.yaml. See the package README for the full parameter reference.

Training

The model was trained and exported with the AWML StreamPETR project:

• https://github.com/tier4/AWML/tree/main/projects/StreamPETR

Training datasets, schedules, and evaluation metrics for these specific weights are not publicly documented.

Limitations

• The model only predicts objects inside the detection range above; the node filters out anything beyond it.
• Only the five classes above are detected. Other road users fall outside the label set.
• The node assumes undistorted input images (or enables undistortion itself), accurate camera calibration, cameras synchronized within max_camera_time_diff, and tf transforms between the camera frames, base_link, and map (for ego motion compensation).
• Predictions do not include object velocity.

Provenance

Original source	https://awf.ml.dev.web.auto/perception/models/streampetr/v1/
This repository	AutowareFoundation/camera_streampetr, tag v1.0

The v1.0 tag corresponds to the streampetr/v1 artifact version at the original hosting path.

Citation

@article{wang2023streampetr,
  title   = {Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection},
  author  = {Wang, Shihao and Liu, Yingfei and Wang, Tiancai and Li, Ying and Zhang, Xiangyu},
  journal = {arXiv preprint arXiv:2303.11926},
  year    = {2023}
}

References

• [1] Wang et al., "Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection", arXiv:2303.11926, 2023.
• AWML StreamPETR project: https://github.com/tier4/AWML/tree/main/projects/StreamPETR
• Autoware: https://github.com/autowarefoundation/autoware