README.md

---
license: apache-2.0
tags:
  - object-detection
  - 3d-object-detection
  - autonomous-driving
  - streampetr
  - carla
  - nuscenes
  - bev
  - tensorrt
  - onnx
datasets:
  - zhijieq/nuCarla
library_name: pytorch
pipeline_tag: object-detection
---

# StreamPETR — nuCarla (Town04)

Camera-only 3D object detection for autonomous driving. Six surround-view
cameras in, 3D bounding boxes out, with a temporal memory carried across
frames. Trained on the [nuCarla](https://huggingface.co/datasets/zhijieq/nuCarla)
dataset, Town04 subset.

Ships as a PyTorch checkpoint plus ready-to-run ONNX graphs and TensorRT FP16
engines.

| | |
|---|---|
| Architecture | [StreamPETR](https://github.com/exiawsh/StreamPETR), ResNet-50 + CPFPN |
| Classes | `car, truck, bus, motorcycle, bicycle, pedestrian` |
| Input | 6 cameras, 1600×900 → 704×256 |
| mAP @0.25 3D IoU | **0.5120** (nuCarla Town04 val) |
| Latency | 4.5 ms/frame, RTX 5070 Ti FP16 (~220 fps) |
| Init | nuScenes-pretrained, class head sliced to 6 classes |

## Demo

`demo/carla_live_detection.mp4` — the model run over a clip recorded live in
CARLA 0.9.16: an ego vehicle driving on autopilot through 150 spawned vehicles
and 16 pedestrians in Town04. Six camera views with projected 3D boxes plus a
bird's-eye panel. Averages 6.7 detections per frame.

The clip is unlabelled, so it is a qualitative check, not a measured score.

## Checkpoints

| File | Size | mAP @0.25 | What it is |
|---|---|---|---|
| `checkpoints/best_mAP_0.25_iter_8512.pth` | 432M | **0.5120** | Best epoch (28). Everything below — ONNX, engines, the demo video, all reported numbers — comes from this one. |
| `checkpoints/final_iter_13376_epoch44.pth` | 432M | 0.4887 | Last checkpoint of the run. mAP had plateaued; kept for continuing training. |
| `checkpoints/init_nuscenes_6class_sliced.pth` | 148M | — | Starting point: the nuScenes StreamPETR weights with the class head sliced from 10 to 6. Use this to reproduce training. |

Intermediate epoch checkpoints are not published — they are rotation artifacts
from `max_keep_ckpts`, not separately evaluated models.

## Files

```
checkpoints/                               the three above
stream_petr_r50_nucarla_town04.py          training config
DEPLOYMENT.md                              full inference guide — read this
demo/carla_live_detection.mp4         12M  detection demo
deployment/
  stream_petr_6cam_encoder.onnx        95M
  stream_petr_6cam_temporal_head.onnx  47M
  *_fp16.engine                             TensorRT 10.9, RTX 5070 Ti
  streampetr_trt_runner.py                  runtime, handles temporal memory
  carla_stream_petr_postprocess.py          decoder
  build_streampetr_engines.py               rebuild engines for another GPU
  test_inputs/, reference_outputs/          one frame, to verify your setup
```

## Quick start (TensorRT)

```bash
pip install tensorrt==10.9.0.34 numpy torch
```

```python
import numpy as np
from streampetr_trt_runner import StreamPETRRunner
from carla_stream_petr_postprocess import decode_stream_petr

runner = StreamPETRRunner(
    "deployment/stream_petr_6cam_encoder_fp16.engine",
    "deployment/stream_petr_6cam_temporal_head_fp16.engine",
)

runner.reset()                       # call at the start of every sequence
for frame in sequence:
    out = runner(
        images=frame.images,         # [1, 6, 3, 256, 704] float32, normalized
        timestamp=frame.seconds,
        ego_pose=frame.lidar2global,       # [1, 4, 4]
        ego_pose_inv=frame.global2lidar,   # [1, 4, 4]
    )
    det = decode_stream_petr(
        out["class_logits"], out["bbox_predictions"], score_threshold=0.35
    )
```

The engines are tied to the GPU and TensorRT version they were built on.
Rebuild from the ONNX for anything else — it takes about 15 seconds:

```bash
python deployment/build_streampetr_engines.py --deployment-dir deployment --cameras 6
```

## Camera rig — must match

The 3D position embedding is **baked into the exported graph**. Images from a
different camera layout produce meaningless geometry with no error raised.
Offsets are ego-frame metres (x forward, y left, z up), yaw counter-clockwise:

| Camera | x | y | z | yaw |
|---|---|---|---|---|
| `CAM_FRONT` | 1.901 | 0.016 | 1.511 | 0.3° |
| `CAM_FRONT_LEFT` | 1.524 | 0.495 | 1.509 | 55.2° |
| `CAM_FRONT_RIGHT` | 1.551 | −0.493 | 1.496 | −56.4° |
| `CAM_BACK` | 0.028 | 0.003 | 1.579 | 179.9° |
| `CAM_BACK_LEFT` | 1.036 | 0.485 | 1.591 | 108.6° |
| `CAM_BACK_RIGHT` | 1.015 | −0.481 | 1.562 | −110.8° |

All six share 1600×900 at 65° horizontal FOV (`fx = fy = 1255.7484`,
`cx = 800`, `cy = 450`) — what CARLA emits for `image_size_x=1600,
image_size_y=900, fov=65`.

Preprocessing, tensor shapes and the temporal memory protocol are all in
`DEPLOYMENT.md`.

## Results

nuCarla Town04 validation, 29 scenes / 1160 frames:

| Class | AP @0.25 IoU |
|---|---|
| car | 0.630 |
| bus | 0.623 |
| truck | 0.612 |
| bicycle | 0.358 |
| motorcycle | 0.317 |
| pedestrian | 0.018 |
| **mAP** | **0.5120** |

Pedestrian scores near zero **under 3D IoU** because the metric is brutal on
small boxes: a 0.42 m wide pedestrian tolerates ~0.25 m of lateral error to
reach IoU 0.25, against ~1.2 m for a car. Under the nuScenes center-distance
metric the paper actually reports, pedestrian AP is **0.309** and overall mAP
**0.4638** — the model does detect pedestrians.

### Not comparable to the nuCarla paper's 0.745

The paper reports mAP 0.745 for PETR, but under different conditions:

| | Paper (PETR) | This model |
|---|---|---|
| Training data | 700 scenes / 7 maps | 91 scenes / **1 map** |
| Backbone | VoVNet | ResNet-50 |
| Resolution | 1600×640 | 704×256 (**5.7× fewer pixels**) |
| GPU-hours | 150 | ~5 |
| Metric | center-distance | 3D IoU |

Town04 is also CARLA's highway map, where objects sit far from the ego (cars
average 50 m away), making it harder than the urban maps in the full set.

## Training

- Initialized from the official nuScenes StreamPETR checkpoint. All six target
  classes exist in nuScenes' ten, so the classification rows were **sliced
  across** rather than randomly re-initialized — nothing in the network starts
  from scratch. This lifted first-epoch mAP from 0.0497 to 0.2701.
- 50 epochs planned, batch 12, AdamW at 1.2e-4 with cosine annealing, FP16.
  The run reached epoch 45; mAP had plateaued since epoch 23 and the LR was
  annealed to 3.6e-6.
- Best checkpoint at iteration 8512 (epoch 28).

## Verification

The ONNX wrapper reproduces the native PyTorch head exactly (`max |Δ| = 0.0`
on both outputs). TensorRT FP16 against PyTorch FP32 on the bundled frame:

| tensor | correlation | rel. error |
|---|---|---|
| `image_features` | 0.999996 | 2.9e-3 |
| `bbox_predictions` | 0.999992 | 2.8e-2 |
| `class_logits` | 0.999546 | 7.6e-2 |

Decoded detections agree: **17 vs 17** above score 0.2, every one placed within
0.02–0.10 m. Two pairs swap rank order, both between detections whose scores
tie to within 0.003.

**Known FP16 effect.** 768 of the 1024 memory slots are bit-exact frame to
frame; within the 256 slots selected fresh each frame, FP16 scoring picks a
slightly different subset near the top-256 cut-off. Its accumulation over long
sequences has not been measured — build the FP32 engine if you need to rule it
out.

## Limitations

- Trained on **one map** (Town04). Expect degradation on urban maps or dense
  intersections.
- Pedestrian localization is weak in absolute terms; suitable for perception
  research, not safety-critical use.
- Simulation only — no real-world images were used, and no sim-to-real transfer
  has been evaluated.
- The rig is fixed. Any change to camera placement, count or intrinsics
  requires re-exporting from the checkpoint.

## Citation

```bibtex
@article{nucarla,
  title={nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation},
  author={Qiao, Zhijie and Cao, Zhong and Liu, Henry X.},
  year={2025},
  url={https://arxiv.org/abs/2511.13744}
}

@article{streampetr,
  title={Exploring Object-Centric Temporal Modeling for Efficient Multi-View 3D Object Detection},
  author={Wang, Shihao and Liu, Yingfei and Wang, Tiancai and Li, Ying and Zhang, Xiangyu},
  journal={arXiv preprint arXiv:2303.11926},
  year={2023}
}
```

## License

Apache 2.0, following [StreamPETR](https://github.com/exiawsh/StreamPETR) and
the nuCarla dataset.