Spaces:
Configuration error
Configuration error
Update README.md
Browse files
README.md
CHANGED
|
@@ -20,11 +20,10 @@ thumbnail: >-
|
|
| 20 |
https://cdn-uploads.huggingface.co/production/uploads/68d15cb11be9adcdaf269758/lTBcnBZVHuqiI2uxDci5D.jpeg
|
| 21 |
---
|
| 22 |
|
| 23 |
-
|
| 24 |
# CARD – Cariad Road Dataset
|
| 25 |
|
| 26 |
-
> **This page is the central index for all CARD dataset releases.**
|
| 27 |
-
> Each sub-dataset has its own repository linked below. The official dataset website
|
| 28 |
|
| 29 |
---
|
| 30 |
|
|
@@ -32,17 +31,18 @@ thumbnail: >-
|
|
| 32 |
|
| 33 |
CARD is a multi-modal driving dataset designed for research in
|
| 34 |
**depth estimation**, **3D reconstruction**, **object detection**, and related
|
| 35 |
-
autonomous-driving tasks. Data was recorded across Germany & Italy
|
| 36 |
-
|
| 37 |
and IMU, yielding rich synchronized multi-modal recordings.
|
| 38 |
|
| 39 |
Key properties:
|
| 40 |
-
- 🎥 **Stereo camera** images (cam_0 / cam_1) at full resolution
|
| 41 |
- 📡 **LiDAR** point clouds + IMU signals
|
| 42 |
- 🏷️ **YOLO-format** bounding-box annotations
|
| 43 |
-
- 📐 **Aggregated depth** point clouds (`agg_depth/`)
|
| 44 |
-
- 🕐 **Temporal consistency**
|
| 45 |
-
- 🔒 **Privacy-preserved**
|
|
|
|
| 46 |
|
| 47 |
---
|
| 48 |
|
|
@@ -56,31 +56,47 @@ Key properties:
|
|
| 56 |
| [CARD-Italy](https://huggingface.co/datasets/CARD-DATA/CARD-Italy) | Italy | 38 | CC BY-NC 4.0 | Non-commercial only |
|
| 57 |
|
| 58 |
> **Note:** Sequences prefixed with `unused_` in any sub-dataset are not part of
|
| 59 |
-
> any official train / val / test split and are provided for zero-shot testing purposes
|
|
|
|
| 60 |
|
| 61 |
---
|
| 62 |
|
| 63 |
-
## Data Format
|
| 64 |
|
| 65 |
Every sequence across all sub-datasets follows the same folder structure:
|
| 66 |
|
| 67 |
```
|
| 68 |
<dataset>/<sequence>/
|
| 69 |
├── img/
|
| 70 |
-
│ ├── cam_0/ #
|
| 71 |
-
│ └── cam_1/ #
|
| 72 |
-
├──
|
|
|
|
| 73 |
├── labels/ # YOLO-format bounding-box annotations
|
| 74 |
-
|
| 75 |
-
└── agg_depth/ # Aggregated depth point clouds (dense GT depth)
|
| 76 |
```
|
| 77 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 78 |
---
|
| 79 |
|
| 80 |
## Splits
|
| 81 |
|
| 82 |
Official **train / val / test** splits are provided as part of the
|
| 83 |
-
[CARD-SDK](https://github.com/cariad-tech/CARD-SDK development kit.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 84 |
|
| 85 |
---
|
| 86 |
|
|
@@ -91,35 +107,810 @@ Official **train / val / test** splits are provided as part of the
|
|
| 91 |
| CARD-Germany-Batch1 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 92 |
| CARD-Germany-Batch2 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 93 |
| CARD-Germany-Batch3 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 94 |
-
| CARD-Italy | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/)
|
| 95 |
|
| 96 |
We have taken reasonable measures to remove personally identifiable information
|
| 97 |
(e.g., faces and license plates). To request removal of specific images from the
|
| 98 |
dataset, please contact **gasser.elazab@cariad.technology**.
|
| 99 |
|
| 100 |
-
The purpose of the CARD project is to help improve road safety
|
| 101 |
-
|
| 102 |
-
use it for any military use.
|
| 103 |
-
|
| 104 |
-
---
|
| 105 |
-
|
| 106 |
-
## Development Kit
|
| 107 |
-
|
| 108 |
-
A development kit (CARD-SDK) with tools to load, visualize, and evaluate on the
|
| 109 |
-
CARD datasets is going to be released soon.
|
| 110 |
|
| 111 |
---
|
| 112 |
|
| 113 |
## Citation
|
| 114 |
|
| 115 |
-
If you use any CARD sub-dataset in your work, please cite
|
| 116 |
|
| 117 |
```bibtex
|
| 118 |
@inproceedings{elazab2025card,
|
| 119 |
-
title={CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction
|
| 120 |
-
|
| 121 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 122 |
year={2026}
|
| 123 |
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 124 |
}
|
| 125 |
-
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 20 |
https://cdn-uploads.huggingface.co/production/uploads/68d15cb11be9adcdaf269758/lTBcnBZVHuqiI2uxDci5D.jpeg
|
| 21 |
---
|
| 22 |
|
|
|
|
| 23 |
# CARD – Cariad Road Dataset
|
| 24 |
|
| 25 |
+
> **This page is the central index for all CARD dataset releases.**
|
| 26 |
+
> Each sub-dataset has its own repository linked below. The official dataset website: http://card.content.cariad.digital/
|
| 27 |
|
| 28 |
---
|
| 29 |
|
|
|
|
| 31 |
|
| 32 |
CARD is a multi-modal driving dataset designed for research in
|
| 33 |
**depth estimation**, **3D reconstruction**, **object detection**, and related
|
| 34 |
+
autonomous-driving tasks. Data was recorded across Germany & Italy
|
| 35 |
+
using a calibrated stereo camera rig paired with 2× LiDAR
|
| 36 |
and IMU, yielding rich synchronized multi-modal recordings.
|
| 37 |
|
| 38 |
Key properties:
|
| 39 |
+
- 🎥 **Stereo camera** images (`cam_0` / `cam_1`) at full resolution
|
| 40 |
- 📡 **LiDAR** point clouds + IMU signals
|
| 41 |
- 🏷️ **YOLO-format** bounding-box annotations
|
| 42 |
+
- 📐 **Aggregated depth** point clouds (`agg_depth/`) — quasi-dense 3D ground truth, processed by aggregation of the LiDAR while removing dynamic-object artifacts
|
| 43 |
+
- 🕐 **Temporal consistency** — all modalities share synchronized microsecond timestamps
|
| 44 |
+
- 🔒 **Privacy-preserved** — faces and license plates anonymized
|
| 45 |
+
- ⛰️ **Challenging topography** — speed bumps, potholes, off-road conditions
|
| 46 |
|
| 47 |
---
|
| 48 |
|
|
|
|
| 56 |
| [CARD-Italy](https://huggingface.co/datasets/CARD-DATA/CARD-Italy) | Italy | 38 | CC BY-NC 4.0 | Non-commercial only |
|
| 57 |
|
| 58 |
> **Note:** Sequences prefixed with `unused_` in any sub-dataset are not part of
|
| 59 |
+
> any official train / val / test split and are provided for zero-shot testing purposes.
|
| 60 |
+
> Sequences with "slam" in their name often involve multiple loops, useful for SLAM evaluations.
|
| 61 |
|
| 62 |
---
|
| 63 |
|
| 64 |
+
## Data Format Overview
|
| 65 |
|
| 66 |
Every sequence across all sub-datasets follows the same folder structure:
|
| 67 |
|
| 68 |
```
|
| 69 |
<dataset>/<sequence>/
|
| 70 |
├── img/
|
| 71 |
+
│ ├── cam_0/ # Right camera images
|
| 72 |
+
│ └── cam_1/ # Left camera images
|
| 73 |
+
├── export/ # Trajectory, calibration, LiDAR, and metadata
|
| 74 |
+
├── agg_depth/ # Aggregated depth point clouds (dense GT depth)
|
| 75 |
├── labels/ # YOLO-format bounding-box annotations
|
| 76 |
+
└── single_depth/ # (optional) single-scan depth
|
|
|
|
| 77 |
```
|
| 78 |
|
| 79 |
+
👉 **For the complete data format specification** — including every JSON schema, coordinate convention, code recipe, and transformation needed to load the data — see the **[Full Data Format Reference](#full-data-format-reference)** below.
|
| 80 |
+
|
| 81 |
+
> **Tip:** You can copy the full reference section below and paste it into any LLM to generate a working data loader, visualization script, or evaluation pipeline from scratch — no SDK required.
|
| 82 |
+
|
| 83 |
---
|
| 84 |
|
| 85 |
## Splits
|
| 86 |
|
| 87 |
Official **train / val / test** splits are provided as part of the
|
| 88 |
+
[CARD-SDK](https://github.com/cariad-tech/CARD-SDK) development kit.
|
| 89 |
+
|
| 90 |
+
---
|
| 91 |
+
|
| 92 |
+
## Development Kit
|
| 93 |
+
|
| 94 |
+
The [CARD-SDK](https://github.com/cariad-tech/CARD-SDK) provides:
|
| 95 |
+
- A PyTorch dataloader with synchronized multi-modal loading
|
| 96 |
+
- A Rerun-based interactive 3D visualizer (`card-viz`)
|
| 97 |
+
- Download utilities for Hugging Face
|
| 98 |
+
|
| 99 |
+
The SDK is **not required** to use the dataset — the full data format reference below is sufficient to write your own loader.
|
| 100 |
|
| 101 |
---
|
| 102 |
|
|
|
|
| 107 |
| CARD-Germany-Batch1 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 108 |
| CARD-Germany-Batch2 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 109 |
| CARD-Germany-Batch3 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
|
| 110 |
+
| CARD-Italy | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — **non-commercial only** |
|
| 111 |
|
| 112 |
We have taken reasonable measures to remove personally identifiable information
|
| 113 |
(e.g., faces and license plates). To request removal of specific images from the
|
| 114 |
dataset, please contact **gasser.elazab@cariad.technology**.
|
| 115 |
|
| 116 |
+
The purpose of the CARD project is to help improve road safety. It is forbidden to
|
| 117 |
+
use this dataset for any military use.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 118 |
|
| 119 |
---
|
| 120 |
|
| 121 |
## Citation
|
| 122 |
|
| 123 |
+
If you use any CARD sub-dataset in your work, please cite:
|
| 124 |
|
| 125 |
```bibtex
|
| 126 |
@inproceedings{elazab2025card,
|
| 127 |
+
title={CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction
|
| 128 |
+
in Challenging Road Topography},
|
| 129 |
+
author={Elazab, Gasser and Neuhaus, Frank and Ko{ss}, Tilman and
|
| 130 |
+
Splietker, Malte and Date, Aditya and Unterreiner, Michael and
|
| 131 |
+
Jansen, Maximilian and Hellwich, Olaf},
|
| 132 |
+
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision
|
| 133 |
+
and Pattern Recognition (CVPR)},
|
| 134 |
year={2026}
|
| 135 |
}
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
---
|
| 139 |
+
---
|
| 140 |
+
|
| 141 |
+
# Full Data Format Reference
|
| 142 |
+
|
| 143 |
+
> **Self-contained specification.** Everything below describes the exact file formats, coordinate systems, JSON schemas, and code needed to load and use CARD data. Copy this entire section into any LLM to generate custom tooling.
|
| 144 |
+
|
| 145 |
+
### Sequence Folder Structure
|
| 146 |
+
|
| 147 |
+
Every sequence across all sub-datasets follows this exact layout:
|
| 148 |
+
|
| 149 |
+
```
|
| 150 |
+
<data_root>/<sequence_name>/
|
| 151 |
+
├── img/
|
| 152 |
+
│ ├── cam_0/ # Right camera images
|
| 153 |
+
│ │ ├── cam_0_<timestamp_us>.jpg
|
| 154 |
+
│ │ ├── cam_0_<timestamp_us>.jpg
|
| 155 |
+
│ │ └── ...
|
| 156 |
+
│ └── cam_1/ # Left camera images
|
| 157 |
+
│ ├── cam_1_<timestamp_us>.jpg
|
| 158 |
+
│ └── ...
|
| 159 |
+
├── export/
|
| 160 |
+
│ ├── output.laz # LiDAR point cloud (compressed LAZ)
|
| 161 |
+
│ ├── output.laz.index.json # Time-indexed access into the LAZ file
|
| 162 |
+
│ ├── output.laz.trajectory.json # Trajectory poses + camera calibration + sensor extrinsics
|
| 163 |
+
│ ├── output.laz.sensor_export.json # IMU + GPS raw data
|
| 164 |
+
│ └── wheel_excitement.json # (optional) wheel excitement analysis
|
| 165 |
+
├── agg_depth/ # Aggregated depth (dense GT)
|
| 166 |
+
│ ├── cam_0/
|
| 167 |
+
│ │ ├── cam_0_<timestamp_us>.npz
|
| 168 |
+
│ │ └── ...
|
| 169 |
+
│ └── cam_1/
|
| 170 |
+
│ ├── cam_1_<timestamp_us>.npz
|
| 171 |
+
│ └── ...
|
| 172 |
+
├── labels/ # YOLO-format bounding-box annotations
|
| 173 |
+
│ ├── cam_0_<timestamp_us>.txt
|
| 174 |
+
│ ├── cam_1_<timestamp_us>.txt
|
| 175 |
+
│ └── ...
|
| 176 |
+
└── single_depth/ # (optional) single-scan depth
|
| 177 |
+
├── cam_0/
|
| 178 |
+
└── cam_1/
|
| 179 |
+
```
|
| 180 |
+
|
| 181 |
+
**Key naming convention:**
|
| 182 |
+
|
| 183 |
+
All timestamps in filenames are in **microseconds** (µs) as integers. For example:
|
| 184 |
+
`cam_0_1697542123456789.jpg` → timestamp = `1697542123456789` µs.
|
| 185 |
+
|
| 186 |
+
### Coordinate Systems & Conventions
|
| 187 |
+
|
| 188 |
+
#### Frame Hierarchy
|
| 189 |
+
|
| 190 |
+
```
|
| 191 |
+
World (inertial, right-handed, Z-up)
|
| 192 |
+
└── Trajectory / Vehicle / Rig Frame (moves with vehicle)
|
| 193 |
+
├── cam_0 (right camera)
|
| 194 |
+
├── cam_1 (left camera)
|
| 195 |
+
├── cariad_wheel_FL_ground
|
| 196 |
+
├── cariad_wheel_FR_ground
|
| 197 |
+
├── cariad_wheel_RL_ground
|
| 198 |
+
└── cariad_wheel_RR_ground
|
| 199 |
+
```
|
| 200 |
+
|
| 201 |
+
#### Axis Conventions
|
| 202 |
+
|
| 203 |
+
| Frame | X | Y | Z | Handedness |
|
| 204 |
+
|---|---|---|---|---|
|
| 205 |
+
| World | East-ish | North-ish | **Up** | Right-handed |
|
| 206 |
+
| Camera | **Right** | **Down** | **Forward** | Right-handed (OpenCV) |
|
| 207 |
+
| Image | **Right** (u) | **Down** (v) | — | Origin top-left |
|
| 208 |
+
|
| 209 |
+
#### Pose Format (7-element vector)
|
| 210 |
+
|
| 211 |
+
All poses in the dataset are stored as:
|
| 212 |
+
```
|
| 213 |
+
[x, y, z, qw, qx, qy, qz]
|
| 214 |
+
```
|
| 215 |
+
- `x, y, z` — translation in **meters**
|
| 216 |
+
- `qw, qx, qy, qz` — unit quaternion, **scalar-first** (Hamilton convention)
|
| 217 |
+
|
| 218 |
+
#### Quaternion ↔ Rotation Matrix
|
| 219 |
+
|
| 220 |
+
```python
|
| 221 |
+
from scipy.spatial.transform import Rotation
|
| 222 |
+
# Given pose = [x, y, z, qw, qx, qy, qz]
|
| 223 |
+
# scipy expects [qx, qy, qz, qw] (scalar-last)
|
| 224 |
+
R = Rotation.from_quat([qx, qy, qz, qw]).as_matrix() # 3×3
|
| 225 |
+
```
|
| 226 |
+
|
| 227 |
+
#### SE(3) Homogeneous Matrix
|
| 228 |
+
|
| 229 |
+
```python
|
| 230 |
+
def pose_to_matrix(x, y, z, qw, qx, qy, qz):
|
| 231 |
+
R = Rotation.from_quat([qx, qy, qz, qw]).as_matrix()
|
| 232 |
+
T = np.eye(4)
|
| 233 |
+
T[:3, :3] = R
|
| 234 |
+
T[:3, 3] = [x, y, z]
|
| 235 |
+
return T # 4×4
|
| 236 |
+
```
|
| 237 |
+
|
| 238 |
+
#### Transform Chain: Camera → World
|
| 239 |
+
|
| 240 |
+
```python
|
| 241 |
+
# 1. Get vehicle pose in world at time t (interpolated from trajectory)
|
| 242 |
+
T_rig_to_world = pose_to_matrix(*interpolate_pose(traj_data, timestamp_us))
|
| 243 |
+
|
| 244 |
+
# 2. Get camera pose in vehicle frame (static, from calibration)
|
| 245 |
+
T_cam_to_rig = pose_to_matrix(*traj_data["sensor_to_trajectory_poses"]["cam_0"])
|
| 246 |
+
|
| 247 |
+
# 3. Chain them
|
| 248 |
+
T_cam_to_world = T_rig_to_world @ T_cam_to_rig
|
| 249 |
+
|
| 250 |
+
# 4. Transform a 3D point from camera coords to world coords:
|
| 251 |
+
pt_world = T_cam_to_world @ [x_cam, y_cam, z_cam, 1.0]
|
| 252 |
+
|
| 253 |
+
# 5. Transform world points to camera coords (for projection):
|
| 254 |
+
T_world_to_cam = np.linalg.inv(T_cam_to_world)
|
| 255 |
+
pt_cam = T_world_to_cam @ [x_world, y_world, z_world, 1.0]
|
| 256 |
+
```
|
| 257 |
+
|
| 258 |
+
**Important:** When looking up the trajectory pose for a camera frame, you must add the camera's `timestamp_offset` to the image timestamp:
|
| 259 |
+
|
| 260 |
+
```python
|
| 261 |
+
trajectory_time = image_timestamp_us + camera_infos[cam_name]["timestamp_offset"]
|
| 262 |
+
pose = interpolate_pose(traj_data, trajectory_time)
|
| 263 |
+
```
|
| 264 |
+
|
| 265 |
+
### Trajectory & Calibration JSON (`export/output.laz.trajectory.json`)
|
| 266 |
+
|
| 267 |
+
This is the single most important file in each sequence. It contains the vehicle trajectory, camera intrinsics, distortion parameters, and all sensor extrinsics.
|
| 268 |
+
|
| 269 |
+
### Full JSON Schema
|
| 270 |
+
|
| 271 |
+
```jsonc
|
| 272 |
+
{
|
| 273 |
+
"trajectory_poses": [
|
| 274 |
+
// Array of [timestamp_us, pose] pairs, sorted by timestamp ascending
|
| 275 |
+
[1697542100000000, [x, y, z, qw, qx, qy, qz]],
|
| 276 |
+
[1697542100050000, [x, y, z, qw, qx, qy, qz]],
|
| 277 |
+
// ... one entry per trajectory sample (high rate)
|
| 278 |
+
],
|
| 279 |
+
|
| 280 |
+
"camera_infos": {
|
| 281 |
+
"cam_0": {
|
| 282 |
+
"fx": 1234.56, // focal length X (pixels)
|
| 283 |
+
"fy": 1234.56, // focal length Y (pixels)
|
| 284 |
+
"cx": 960.0, // principal point X (pixels)
|
| 285 |
+
"cy": 540.0, // principal point Y (pixels)
|
| 286 |
+
"image_width": 1920, // image width (pixels)
|
| 287 |
+
"image_height": 1080, // image height (pixels)
|
| 288 |
+
"dist_coeffs": [k1, k2, p1, p2, k3], // OpenCV Brown distortion model (5 params)
|
| 289 |
+
"timestamp_offset": 12345 // offset in microseconds to align camera time to trajectory time
|
| 290 |
+
},
|
| 291 |
+
"cam_1": {
|
| 292 |
+
// same structure
|
| 293 |
+
}
|
| 294 |
+
},
|
| 295 |
+
|
| 296 |
+
"sensor_to_trajectory_poses": {
|
| 297 |
+
// Static transforms: each sensor's pose relative to the vehicle/rig frame
|
| 298 |
+
"cam_0": [x, y, z, qw, qx, qy, qz],
|
| 299 |
+
"cam_1": [x, y, z, qw, qx, qy, qz],
|
| 300 |
+
"cariad_wheel_FL_ground": [x, y, z, qw, qx, qy, qz], // note: capital G variant "Ground" may also appear
|
| 301 |
+
"cariad_wheel_FR_ground": [x, y, z, qw, qx, qy, qz],
|
| 302 |
+
"cariad_wheel_RL_ground": [x, y, z, qw, qx, qy, qz],
|
| 303 |
+
"cariad_wheel_RR_ground": [x, y, z, qw, qx, qy, qz]
|
| 304 |
+
// ... may contain additional sensor frames
|
| 305 |
+
}
|
| 306 |
+
}
|
| 307 |
+
```
|
| 308 |
+
|
| 309 |
+
### Camera Intrinsics Matrix (K)
|
| 310 |
+
|
| 311 |
+
```
|
| 312 |
+
K = | fx 0 cx |
|
| 313 |
+
| 0 fy cy |
|
| 314 |
+
| 0 0 1 |
|
| 315 |
+
```
|
| 316 |
+
|
| 317 |
+
### Distortion Model
|
| 318 |
+
|
| 319 |
+
OpenCV Brown's 5-parameter model: `[k1, k2, p1, p2, k3]`
|
| 320 |
+
- `k1, k2, k3` — radial distortion coefficients
|
| 321 |
+
- `p1, p2` — tangential distortion coefficients
|
| 322 |
+
|
| 323 |
+
**Undistortion:**
|
| 324 |
+
```python
|
| 325 |
+
import cv2
|
| 326 |
+
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
|
| 327 |
+
dist = np.array(camera_info["dist_coeffs"])
|
| 328 |
+
|
| 329 |
+
# Simple approach:
|
| 330 |
+
undistorted = cv2.undistort(image, K, dist)
|
| 331 |
+
|
| 332 |
+
# Better approach (computes optimal new intrinsics):
|
| 333 |
+
newK, roi = cv2.getOptimalNewCameraMatrix(K, dist, (W, H), alpha=0)
|
| 334 |
+
mapx, mapy = cv2.initUndistortRectifyMap(K, dist, None, newK, (W, H), cv2.CV_32FC1)
|
| 335 |
+
undistorted = cv2.remap(image, mapx, mapy, cv2.INTER_LINEAR)
|
| 336 |
+
# Use newK for all subsequent projection operations on undistorted images
|
| 337 |
+
```
|
| 338 |
+
|
| 339 |
+
### Pose Interpolation
|
| 340 |
+
|
| 341 |
+
Trajectory poses are sampled at discrete times. To get the pose at an arbitrary timestamp:
|
| 342 |
+
|
| 343 |
+
```python
|
| 344 |
+
def interpolate_pose(traj_data, timestamp_us):
|
| 345 |
+
poses = traj_data["trajectory_poses"]
|
| 346 |
+
# Find i such that poses[i-1][0] <= timestamp_us < poses[i][0]
|
| 347 |
+
i = 0
|
| 348 |
+
while i < len(poses) and poses[i][0] <= timestamp_us:
|
| 349 |
+
i += 1
|
| 350 |
+
|
| 351 |
+
if i == 0:
|
| 352 |
+
return poses[0][1] # before first pose: clamp
|
| 353 |
+
if i >= len(poses):
|
| 354 |
+
return poses[-1][1] # after last pose: clamp
|
| 355 |
+
|
| 356 |
+
ts0, pose0 = poses[i - 1]
|
| 357 |
+
ts1, pose1 = poses[i]
|
| 358 |
+
t = (timestamp_us - ts0) / (ts1 - ts0) # interpolation factor [0, 1]
|
| 359 |
+
|
| 360 |
+
# Linear interpolation of translation
|
| 361 |
+
trans = (1 - t) * np.array(pose0[:3]) + t * np.array(pose1[:3])
|
| 362 |
+
|
| 363 |
+
# SLERP for rotation (quaternion scalar-first → convert to scipy scalar-last)
|
| 364 |
+
from scipy.spatial.transform import Rotation, Slerp
|
| 365 |
+
q0_xyzw = [pose0[4], pose0[5], pose0[6], pose0[3]] # qx,qy,qz,qw
|
| 366 |
+
q1_xyzw = [pose1[4], pose1[5], pose1[6], pose1[3]]
|
| 367 |
+
rotations = Rotation.from_quat([q0_xyzw, q1_xyzw])
|
| 368 |
+
slerp = Slerp([0.0, 1.0], rotations)
|
| 369 |
+
q_interp_xyzw = slerp([t]).as_quat()[0]
|
| 370 |
+
|
| 371 |
+
# Back to scalar-first
|
| 372 |
+
qw = q_interp_xyzw[3]
|
| 373 |
+
qx, qy, qz = q_interp_xyzw[:3]
|
| 374 |
+
return [trans[0], trans[1], trans[2], qw, qx, qy, qz]
|
| 375 |
+
```
|
| 376 |
+
|
| 377 |
+
### Camera Images
|
| 378 |
+
|
| 379 |
+
### Location & Naming
|
| 380 |
+
|
| 381 |
+
```
|
| 382 |
+
<sequence>/img/cam_0/cam_0_<timestamp_us>.jpg # right camera
|
| 383 |
+
<sequence>/img/cam_1/cam_1_<timestamp_us>.jpg # left camera
|
| 384 |
+
```
|
| 385 |
+
|
| 386 |
+
- `cam_1` = **left** camera, `cam_0` = **right** camera (stereo pair)
|
| 387 |
+
- Filename regex: `cam_\d+_(\d+)\.(?:jpg|jpeg|png)` where group 1 is the timestamp in microseconds
|
| 388 |
+
- Images are stored in their **original (distorted)** form — you must undistort using the calibration
|
| 389 |
+
|
| 390 |
+
### Loading & Undistortion
|
| 391 |
+
|
| 392 |
+
```python
|
| 393 |
+
import cv2
|
| 394 |
+
import numpy as np
|
| 395 |
+
|
| 396 |
+
# Load image (OpenCV reads as BGR)
|
| 397 |
+
image = cv2.imread("cam_0_1697542123456789.jpg")
|
| 398 |
+
|
| 399 |
+
# Get calibration from trajectory JSON
|
| 400 |
+
cam_info = traj_data["camera_infos"]["cam_0"]
|
| 401 |
+
fx, fy = cam_info["fx"], cam_info["fy"]
|
| 402 |
+
cx, cy = cam_info["cx"], cam_info["cy"]
|
| 403 |
+
K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
|
| 404 |
+
dist = np.array(cam_info["dist_coeffs"])
|
| 405 |
+
|
| 406 |
+
# Undistort
|
| 407 |
+
undistorted = cv2.undistort(image, K, dist)
|
| 408 |
+
```
|
| 409 |
+
|
| 410 |
+
### Stereo Pair Convention
|
| 411 |
+
|
| 412 |
+
| Name | Role | Position |
|
| 413 |
+
|---|---|---|
|
| 414 |
+
| `cam_1` | Left camera | Left on vehicle |
|
| 415 |
+
| `cam_0` | Right camera | Right on vehicle |
|
| 416 |
+
|
| 417 |
+
Stereo timestamps are synchronized to within ~2 ms (2000 µs tolerance).
|
| 418 |
+
|
| 419 |
+
### Stereo Rectification
|
| 420 |
+
|
| 421 |
+
```python
|
| 422 |
+
# Get relative transform: left → right
|
| 423 |
+
pose_left = traj_data["sensor_to_trajectory_poses"]["cam_1"] # [x,y,z,qw,qx,qy,qz]
|
| 424 |
+
pose_right = traj_data["sensor_to_trajectory_poses"]["cam_0"]
|
| 425 |
+
|
| 426 |
+
T_left = pose_to_matrix(*pose_left)
|
| 427 |
+
T_right = pose_to_matrix(*pose_right)
|
| 428 |
+
T_right_from_left = np.linalg.inv(T_right) @ T_left
|
| 429 |
+
R = T_right_from_left[:3, :3]
|
| 430 |
+
t = T_right_from_left[:3, 3]
|
| 431 |
+
|
| 432 |
+
# Rectify
|
| 433 |
+
image_size = (cam_info["image_width"], cam_info["image_height"])
|
| 434 |
+
R1, R2, P1, P2, Q, _, _ = cv2.stereoRectify(
|
| 435 |
+
K_left, dist_left, K_right, dist_right, image_size, R, t,
|
| 436 |
+
flags=cv2.CALIB_ZERO_DISPARITY
|
| 437 |
+
)
|
| 438 |
+
|
| 439 |
+
# Build rectification maps
|
| 440 |
+
map1_l, map2_l = cv2.initUndistortRectifyMap(K_left, dist_left, R1, P1, image_size, cv2.CV_16SC2)
|
| 441 |
+
map1_r, map2_r = cv2.initUndistortRectifyMap(K_right, dist_right, R2, P2, image_size, cv2.CV_16SC2)
|
| 442 |
+
|
| 443 |
+
# Apply
|
| 444 |
+
rect_left = cv2.remap(img_left, map1_l, map2_l, cv2.INTER_LINEAR)
|
| 445 |
+
rect_right = cv2.remap(img_right, map1_r, map2_r, cv2.INTER_LINEAR)
|
| 446 |
+
|
| 447 |
+
# Baseline in meters:
|
| 448 |
+
baseline_m = np.linalg.norm(t)
|
| 449 |
+
```
|
| 450 |
+
|
| 451 |
+
### LiDAR Point Clouds (LAZ)
|
| 452 |
+
|
| 453 |
+
### Files
|
| 454 |
+
|
| 455 |
+
```
|
| 456 |
+
<sequence>/export/output.laz # compressed LiDAR data (all scans)
|
| 457 |
+
<sequence>/export/output.laz.index.json # time-based index for random access
|
| 458 |
+
```
|
| 459 |
+
|
| 460 |
+
### Dependencies
|
| 461 |
+
|
| 462 |
+
```bash
|
| 463 |
+
pip install "laspy==2.6.1" "lazrs==0.8.0"
|
| 464 |
+
```
|
| 465 |
+
|
| 466 |
+
### Index JSON Schema
|
| 467 |
+
|
| 468 |
+
```jsonc
|
| 469 |
+
{
|
| 470 |
+
"entries": [
|
| 471 |
+
{
|
| 472 |
+
"end_ts": 1697542100050000, // end timestamp of this chunk (microseconds)
|
| 473 |
+
"point_offset_in_file": 0, // point offset (NOT byte offset) into LAZ file
|
| 474 |
+
"num_pts": 5000, // number of points in this chunk
|
| 475 |
+
"laser_name": "lidar_0", // which laser scanner produced this chunk
|
| 476 |
+
"part_index": 0 // always 0 (multipart not supported)
|
| 477 |
+
},
|
| 478 |
+
// ... entries sorted by end_ts ascending
|
| 479 |
+
]
|
| 480 |
+
}
|
| 481 |
+
```
|
| 482 |
+
|
| 483 |
+
### Reading LiDAR Points
|
| 484 |
+
|
| 485 |
+
```python
|
| 486 |
+
import laspy
|
| 487 |
+
import json
|
| 488 |
+
import numpy as np
|
| 489 |
+
|
| 490 |
+
# Load the index
|
| 491 |
+
with open("export/output.laz.index.json") as f:
|
| 492 |
+
index = json.load(f)["entries"]
|
| 493 |
+
index.sort(key=lambda e: e["end_ts"])
|
| 494 |
+
|
| 495 |
+
# Read a specific chunk
|
| 496 |
+
with laspy.open("export/output.laz") as las:
|
| 497 |
+
entry = index[42] # pick an entry
|
| 498 |
+
las.seek(entry["point_offset_in_file"])
|
| 499 |
+
points = las.read_points(entry["num_pts"])
|
| 500 |
+
|
| 501 |
+
# Extract coordinates (world frame, meters)
|
| 502 |
+
coords = np.vstack((points.x, points.y, points.z)).T # shape (N, 3)
|
| 503 |
+
|
| 504 |
+
# Optional: intensity (normalized 0–1)
|
| 505 |
+
if hasattr(points, "intensity"):
|
| 506 |
+
intensity = points.intensity / 65535.0
|
| 507 |
+
```
|
| 508 |
+
|
| 509 |
+
### Time-Based Query
|
| 510 |
+
|
| 511 |
+
To get all LiDAR points in a time window `[t0, t1]` (in microseconds):
|
| 512 |
+
|
| 513 |
+
```python
|
| 514 |
+
import bisect
|
| 515 |
+
|
| 516 |
+
end_times = [e["end_ts"] for e in index]
|
| 517 |
+
|
| 518 |
+
# Select entries with end_ts in (t0, t1]
|
| 519 |
+
lo = bisect.bisect_right(end_times, t0)
|
| 520 |
+
hi = bisect.bisect_right(end_times, t1)
|
| 521 |
+
selected_entries = index[lo:hi]
|
| 522 |
+
|
| 523 |
+
# Read all selected chunks
|
| 524 |
+
all_points = []
|
| 525 |
+
with laspy.open("export/output.laz") as las:
|
| 526 |
+
for entry in selected_entries:
|
| 527 |
+
las.seek(entry["point_offset_in_file"])
|
| 528 |
+
pts = las.read_points(entry["num_pts"])
|
| 529 |
+
all_points.append(np.vstack((pts.x, pts.y, pts.z)).T)
|
| 530 |
+
|
| 531 |
+
points_world = np.concatenate(all_points, axis=0) # shape (N, 3) in world coords
|
| 532 |
+
```
|
| 533 |
+
|
| 534 |
+
### LiDAR Coordinate Frame
|
| 535 |
+
|
| 536 |
+
LiDAR points in the LAZ file are in **world coordinates** (same frame as trajectory poses). To project them into a camera:
|
| 537 |
+
|
| 538 |
+
```python
|
| 539 |
+
# 1. Get camera pose at time t
|
| 540 |
+
T_cam_to_world = T_rig_to_world @ T_cam_to_rig # see "Transform Chain" above
|
| 541 |
+
T_world_to_cam = np.linalg.inv(T_cam_to_world)
|
| 542 |
+
|
| 543 |
+
# 2. Transform points
|
| 544 |
+
pts_homogeneous = np.c_[points_world, np.ones(len(points_world))]
|
| 545 |
+
pts_cam = (T_world_to_cam @ pts_homogeneous.T).T[:, :3]
|
| 546 |
+
|
| 547 |
+
# 3. Project to image (only points in front of camera)
|
| 548 |
+
z = pts_cam[:, 2]
|
| 549 |
+
valid = z > 0
|
| 550 |
+
x, y, z = pts_cam[valid, 0], pts_cam[valid, 1], pts_cam[valid, 2]
|
| 551 |
+
u = (fx * x / z) + cx
|
| 552 |
+
v = (fy * y / z) + cy
|
| 553 |
+
|
| 554 |
+
# 4. Filter to image bounds
|
| 555 |
+
in_bounds = (u >= 0) & (u < W) & (v >= 0) & (v < H)
|
| 556 |
+
```
|
| 557 |
+
|
| 558 |
+
### Aggregated Depth (Dense Ground Truth)
|
| 559 |
+
|
| 560 |
+
Aggregated depth files contain quasi-dense 3D point clouds produced by accumulating many LiDAR scans over time and removing dynamic-object artifacts. This is the primary depth ground truth.
|
| 561 |
+
|
| 562 |
+
### Location & Naming
|
| 563 |
+
|
| 564 |
+
```
|
| 565 |
+
<sequence>/agg_depth/cam_0/cam_0_<timestamp_us>.npz
|
| 566 |
+
<sequence>/agg_depth/cam_1/cam_1_<timestamp_us>.npz
|
| 567 |
+
```
|
| 568 |
+
|
| 569 |
+
Each `.npz` file corresponds to one camera frame (matched by timestamp).
|
| 570 |
+
|
| 571 |
+
### NPZ Contents
|
| 572 |
+
|
| 573 |
+
| Key | Shape | Type | Description |
|
| 574 |
+
|---|---|---|---|
|
| 575 |
+
| `pts_cam` | `(N, 3)` | `float32` | 3D points in **camera frame** coordinates |
|
| 576 |
+
|
| 577 |
+
Camera frame coordinates: X = right, Y = down, Z = forward.
|
| 578 |
+
|
| 579 |
+
### Loading & Projecting to Depth Map
|
| 580 |
+
|
| 581 |
+
```python
|
| 582 |
+
import numpy as np
|
| 583 |
+
|
| 584 |
+
data = np.load("agg_depth/cam_0/cam_0_1697542123456789.npz")
|
| 585 |
+
pts_cam = data["pts_cam"].astype(np.float32) # (N, 3) in camera coords
|
| 586 |
+
|
| 587 |
+
# Project to depth map
|
| 588 |
+
H, W = camera_info["image_height"], camera_info["image_width"]
|
| 589 |
+
depth_map = np.zeros((H, W), dtype=np.float32)
|
| 590 |
+
|
| 591 |
+
z = pts_cam[:, 2]
|
| 592 |
+
valid = z > 0 # keep points in front of camera
|
| 593 |
+
x, y, z = pts_cam[valid, 0], pts_cam[valid, 1], pts_cam[valid, 2]
|
| 594 |
+
|
| 595 |
+
u = (fx * x / z) + cx # project to pixel coords
|
| 596 |
+
v = (fy * y / z) + cy
|
| 597 |
+
|
| 598 |
+
in_bounds = (u >= 0) & (u < W) & (v >= 0) & (v < H)
|
| 599 |
+
u, v, z = u[in_bounds], v[in_bounds], z[in_bounds]
|
| 600 |
+
|
| 601 |
+
# Z-buffer: sort far-to-near so near points overwrite far ones
|
| 602 |
+
order = np.argsort(z)[::-1]
|
| 603 |
+
u, v, z = u[order], v[order], z[order]
|
| 604 |
+
|
| 605 |
+
depth_map[v.astype(int), u.astype(int)] = z
|
| 606 |
+
# depth_map is now (H, W) with depth in meters (0 = no data)
|
| 607 |
+
```
|
| 608 |
+
|
| 609 |
+
### Transforming to World Coordinates
|
| 610 |
+
|
| 611 |
+
```python
|
| 612 |
+
# If you need world coordinates from agg_depth pts_cam:
|
| 613 |
+
T_cam_to_rig = pose_to_matrix(*traj_data["sensor_to_trajectory_poses"]["cam_0"])
|
| 614 |
+
pose_rig = interpolate_pose(traj_data, timestamp_us + camera_info["timestamp_offset"])
|
| 615 |
+
T_rig_to_world = pose_to_matrix(*pose_rig)
|
| 616 |
+
T_cam_to_world = T_rig_to_world @ T_cam_to_rig
|
| 617 |
+
|
| 618 |
+
pts_world = (T_cam_to_world @ np.c_[pts_cam, np.ones(len(pts_cam))].T).T[:, :3]
|
| 619 |
+
```
|
| 620 |
+
|
| 621 |
+
### Single-Frame Depth
|
| 622 |
+
|
| 623 |
+
Some sequences include single-scan LiDAR depth (one frame, not aggregated).
|
| 624 |
+
|
| 625 |
+
### Location
|
| 626 |
+
|
| 627 |
+
```
|
| 628 |
+
<sequence>/single_depth/<cam_name>/<cam_name>_<timestamp_us>.npz
|
| 629 |
+
```
|
| 630 |
+
|
| 631 |
+
### NPZ Contents
|
| 632 |
+
|
| 633 |
+
May contain either or both:
|
| 634 |
+
|
| 635 |
+
| Key | Shape | Type | Description |
|
| 636 |
+
|---|---|---|---|
|
| 637 |
+
| `pts_cam` | `(N, 3)` | `float32` | 3D points in camera frame |
|
| 638 |
+
| `depth` | `(H, W)` | `float32` | Pre-projected depth map |
|
| 639 |
+
|
| 640 |
+
Loading logic: use `depth` if present, otherwise project `pts_cam` to a depth map using the same projection formula as aggregated depth.
|
| 641 |
+
|
| 642 |
+
Alternative directory names may be present:
|
| 643 |
+
- `single_lidar/<cam>/`, `lidar_single/<cam>/`, `sparse_lidar/<cam>/`, `lidar_sparse/<cam>/`
|
| 644 |
+
|
| 645 |
+
### YOLO Bounding-Box Labels
|
| 646 |
+
|
| 647 |
+
### Location & Naming
|
| 648 |
+
|
| 649 |
+
```
|
| 650 |
+
<sequence>/labels/<cam_name>_<timestamp_us>.txt
|
| 651 |
+
```
|
| 652 |
+
|
| 653 |
+
Note: labels are in a flat `labels/` directory (not per-camera subdirectories).
|
| 654 |
+
|
| 655 |
+
### Format
|
| 656 |
+
|
| 657 |
+
One object per line, space-separated:
|
| 658 |
+
```
|
| 659 |
+
<class_id> <x_center_norm> <y_center_norm> <width_norm> <height_norm>
|
| 660 |
+
```
|
| 661 |
+
|
| 662 |
+
All values are **normalized** to `[0, 1]` relative to image dimensions.
|
| 663 |
+
|
| 664 |
+
### Conversion to Pixel Coordinates
|
| 665 |
+
|
| 666 |
+
```python
|
| 667 |
+
# Read labels
|
| 668 |
+
H, W = image.shape[:2] # original image dimensions
|
| 669 |
+
with open(label_file) as f:
|
| 670 |
+
for line in f:
|
| 671 |
+
cls, xc, yc, bw, bh = map(float, line.strip().split())
|
| 672 |
+
cls = int(cls)
|
| 673 |
+
|
| 674 |
+
# Convert to pixel coordinates
|
| 675 |
+
x_center_px = xc * W
|
| 676 |
+
y_center_px = yc * H
|
| 677 |
+
width_px = bw * W
|
| 678 |
+
height_px = bh * H
|
| 679 |
+
|
| 680 |
+
# Bounding box corners (top-left, bottom-right)
|
| 681 |
+
x1 = int(x_center_px - width_px / 2)
|
| 682 |
+
y1 = int(y_center_px - height_px / 2)
|
| 683 |
+
x2 = int(x_center_px + width_px / 2)
|
| 684 |
+
y2 = int(y_center_px + height_px / 2)
|
| 685 |
+
```
|
| 686 |
+
|
| 687 |
+
### IMU & GPS Data
|
| 688 |
+
|
| 689 |
+
### File
|
| 690 |
+
|
| 691 |
+
```
|
| 692 |
+
<sequence>/export/output.laz.sensor_export.json
|
| 693 |
+
```
|
| 694 |
+
|
| 695 |
+
### JSON Schema
|
| 696 |
+
|
| 697 |
+
```jsonc
|
| 698 |
+
{
|
| 699 |
+
"imu": [
|
| 700 |
+
{
|
| 701 |
+
"timestamp": 1697542100000000, // microseconds
|
| 702 |
+
"accel_x": 0.12, // m/s² — X acceleration
|
| 703 |
+
"accel_y": -0.03, // m/s² — Y acceleration
|
| 704 |
+
"accel_z": 9.81, // m/s² — Z acceleration
|
| 705 |
+
"gyro_x": 0.001, // rad/s — X angular velocity
|
| 706 |
+
"gyro_y": -0.002, // rad/s — Y angular velocity
|
| 707 |
+
"gyro_z": 0.0005 // rad/s — Z angular velocity
|
| 708 |
+
},
|
| 709 |
+
// ...
|
| 710 |
+
],
|
| 711 |
+
|
| 712 |
+
"gps": [
|
| 713 |
+
{
|
| 714 |
+
"timestamp": 1697542100000000, // system timestamp (microseconds)
|
| 715 |
+
"gps_timestamp_us": 1697542100012345, // GPS receiver timestamp (microseconds)
|
| 716 |
+
"lat_rad": 0.8527, // latitude in RADIANS
|
| 717 |
+
"lon_rad": 0.1658 // longitude in RADIANS
|
| 718 |
+
},
|
| 719 |
+
// ...
|
| 720 |
+
]
|
| 721 |
+
}
|
| 722 |
+
```
|
| 723 |
+
|
| 724 |
+
### GPS Conversion
|
| 725 |
+
|
| 726 |
+
```python
|
| 727 |
+
# Radians to degrees
|
| 728 |
+
lat_deg = gps_entry["lat_rad"] / np.pi * 180.0
|
| 729 |
+
lon_deg = gps_entry["lon_rad"] / np.pi * 180.0
|
| 730 |
+
```
|
| 731 |
+
|
| 732 |
+
### IMU Frame
|
| 733 |
+
|
| 734 |
+
IMU measurements are in the vehicle/rig frame (axes aligned with the vehicle body).
|
| 735 |
+
|
| 736 |
+
### Wheel Excitement Data
|
| 737 |
+
|
| 738 |
+
Optional file present in some sequences.
|
| 739 |
+
|
| 740 |
+
### File
|
| 741 |
+
|
| 742 |
+
```
|
| 743 |
+
<sequence>/export/wheel_excitement.json
|
| 744 |
+
```
|
| 745 |
+
|
| 746 |
+
### JSON Schema
|
| 747 |
+
|
| 748 |
+
```jsonc
|
| 749 |
+
{
|
| 750 |
+
"wheel_results": {
|
| 751 |
+
"<frame_id>": { // e.g., "sensor_wheel_FL"
|
| 752 |
+
"wheel_excitement": [
|
| 753 |
+
[timestamp_us, excitement_value], // float pairs
|
| 754 |
+
// ...
|
| 755 |
+
],
|
| 756 |
+
"contact_point_trajectory": { // (optional)
|
| 757 |
+
"poses": [
|
| 758 |
+
[timestamp_us, [x, y, z, qw, qx, qy, qz]],
|
| 759 |
+
// ...
|
| 760 |
+
]
|
| 761 |
+
},
|
| 762 |
+
"ground_trajectory": { // (optional, alternative to contact_point_trajectory)
|
| 763 |
+
"poses": [
|
| 764 |
+
[timestamp_us, [x, y, z, qw, qx, qy, qz]],
|
| 765 |
+
// ...
|
| 766 |
+
]
|
| 767 |
+
}
|
| 768 |
+
},
|
| 769 |
+
// ... one entry per wheel (FL, FR, RL, RR)
|
| 770 |
+
}
|
| 771 |
}
|
| 772 |
+
```
|
| 773 |
+
|
| 774 |
+
The wheel identifier (FL, FR, RL, RR) can be extracted from the frame_id by taking the last two characters after splitting by `_`.
|
| 775 |
+
|
| 776 |
+
### Sensor Extrinsics & Ground Plane
|
| 777 |
+
|
| 778 |
+
### Sensor Extrinsics
|
| 779 |
+
|
| 780 |
+
All sensor poses relative to the vehicle frame are in `sensor_to_trajectory_poses`:
|
| 781 |
+
|
| 782 |
+
```python
|
| 783 |
+
# Camera extrinsic (camera → vehicle frame)
|
| 784 |
+
cam_pose = traj_data["sensor_to_trajectory_poses"]["cam_0"] # [x,y,z,qw,qx,qy,qz]
|
| 785 |
+
T_cam_to_rig = pose_to_matrix(*cam_pose)
|
| 786 |
+
```
|
| 787 |
+
|
| 788 |
+
### Ground Plane from Wheel Contact Points
|
| 789 |
+
|
| 790 |
+
The four wheel ground contact points define the road plane beneath the vehicle:
|
| 791 |
+
|
| 792 |
+
```python
|
| 793 |
+
wheel_keys = [
|
| 794 |
+
"cariad_wheel_FL_ground", # or "cariad_wheel_FL_Ground"
|
| 795 |
+
"cariad_wheel_FR_ground",
|
| 796 |
+
"cariad_wheel_RL_ground",
|
| 797 |
+
"cariad_wheel_RR_ground",
|
| 798 |
+
]
|
| 799 |
+
|
| 800 |
+
# Extract wheel positions (in vehicle frame)
|
| 801 |
+
wheel_points = []
|
| 802 |
+
for key in wheel_keys:
|
| 803 |
+
if key in traj_data["sensor_to_trajectory_poses"]:
|
| 804 |
+
pose = traj_data["sensor_to_trajectory_poses"][key]
|
| 805 |
+
wheel_points.append([pose[0], pose[1], pose[2]])
|
| 806 |
+
|
| 807 |
+
wheel_points = np.array(wheel_points) # (4, 3)
|
| 808 |
+
centroid = np.mean(wheel_points, axis=0)
|
| 809 |
+
|
| 810 |
+
# Fit plane via SVD
|
| 811 |
+
_, _, vh = np.linalg.svd(wheel_points - centroid)
|
| 812 |
+
normal = vh[-1] # plane normal
|
| 813 |
+
if normal[2] < 0:
|
| 814 |
+
normal = -normal # ensure "up" direction
|
| 815 |
+
|
| 816 |
+
# Ground plane: normal · (p - centroid) = 0
|
| 817 |
+
# Typical centroid z ≈ -1.8m (ground is ~1.8m below vehicle center)
|
| 818 |
+
```
|
| 819 |
+
|
| 820 |
+
### How To: Core Recipes
|
| 821 |
+
|
| 822 |
+
#### Recipe 1: Load a stereo pair with depth ground truth
|
| 823 |
+
|
| 824 |
+
```python
|
| 825 |
+
import json, cv2, numpy as np, re, os
|
| 826 |
+
|
| 827 |
+
seq_dir = "/path/to/sequence"
|
| 828 |
+
|
| 829 |
+
# 1. Load calibration
|
| 830 |
+
with open(os.path.join(seq_dir, "export/output.laz.trajectory.json")) as f:
|
| 831 |
+
traj = json.load(f)
|
| 832 |
+
|
| 833 |
+
# 2. Pick a timestamp (e.g., first image)
|
| 834 |
+
cam = "cam_1" # left camera
|
| 835 |
+
cam_dir = os.path.join(seq_dir, "img", cam)
|
| 836 |
+
files = sorted(os.listdir(cam_dir))
|
| 837 |
+
timestamp = int(re.match(r"cam_\d+_(\d+)\.jpg", files[0]).group(1))
|
| 838 |
+
|
| 839 |
+
# 3. Load & undistort image
|
| 840 |
+
ci = traj["camera_infos"][cam]
|
| 841 |
+
K = np.array([[ci["fx"], 0, ci["cx"]], [0, ci["fy"], ci["cy"]], [0, 0, 1]])
|
| 842 |
+
dist = np.array(ci["dist_coeffs"])
|
| 843 |
+
img = cv2.undistort(cv2.imread(os.path.join(cam_dir, files[0])), K, dist)
|
| 844 |
+
|
| 845 |
+
# 4. Load aggregated depth
|
| 846 |
+
npz = np.load(os.path.join(seq_dir, f"agg_depth/{cam}/{cam}_{timestamp}.npz"))
|
| 847 |
+
pts_cam = npz["pts_cam"].astype(np.float32)
|
| 848 |
+
|
| 849 |
+
# 5. Project to depth map
|
| 850 |
+
H, W = ci["image_height"], ci["image_width"]
|
| 851 |
+
# ... (use projection code from "Aggregated Depth" section)
|
| 852 |
+
```
|
| 853 |
+
|
| 854 |
+
#### Recipe 2: Get LiDAR points for a camera frame
|
| 855 |
+
|
| 856 |
+
```python
|
| 857 |
+
import bisect
|
| 858 |
+
|
| 859 |
+
# Camera timestamp → trajectory time
|
| 860 |
+
traj_time = timestamp + ci["timestamp_offset"]
|
| 861 |
+
|
| 862 |
+
# Find LAZ entries near this time (e.g., ±50ms window)
|
| 863 |
+
with open(os.path.join(seq_dir, "export/output.laz.index.json")) as f:
|
| 864 |
+
index = json.load(f)["entries"]
|
| 865 |
+
index.sort(key=lambda e: e["end_ts"])
|
| 866 |
+
end_times = [e["end_ts"] for e in index]
|
| 867 |
+
|
| 868 |
+
window_us = 100_000 # 100ms
|
| 869 |
+
t0 = traj_time - window_us // 2
|
| 870 |
+
t1 = traj_time + window_us // 2
|
| 871 |
+
lo = bisect.bisect_right(end_times, t0)
|
| 872 |
+
hi = bisect.bisect_right(end_times, t1)
|
| 873 |
+
entries = index[lo:hi]
|
| 874 |
+
|
| 875 |
+
# Read and project (see LiDAR section for full code)
|
| 876 |
+
```
|
| 877 |
+
|
| 878 |
+
#### Recipe 3: Relative pose between two frames
|
| 879 |
+
|
| 880 |
+
```python
|
| 881 |
+
# Useful for multi-view depth estimation, visual odometry, etc.
|
| 882 |
+
T_cam_world_t0 = T_rig_world_t0 @ T_cam_rig
|
| 883 |
+
T_cam_world_t1 = T_rig_world_t1 @ T_cam_rig
|
| 884 |
+
|
| 885 |
+
# Relative pose: t1 expressed in t0's camera frame
|
| 886 |
+
T_t1_in_t0 = np.linalg.inv(T_cam_world_t0) @ T_cam_world_t1
|
| 887 |
+
```
|
| 888 |
+
|
| 889 |
+
### Split Files
|
| 890 |
+
|
| 891 |
+
Official **train / val / test** splits are plain text files with one sample per line:
|
| 892 |
+
|
| 893 |
+
```
|
| 894 |
+
<sequence_name>/img/<cam_name>/<cam_name>_<timestamp_us>.jpg
|
| 895 |
+
```
|
| 896 |
+
|
| 897 |
+
Example:
|
| 898 |
+
```
|
| 899 |
+
sequence_001/img/cam_0/cam_0_1697542123456789.jpg
|
| 900 |
+
sequence_001/img/cam_0/cam_0_1697542123556789.jpg
|
| 901 |
+
sequence_002/img/cam_1/cam_1_1697543200000000.jpg
|
| 902 |
+
```
|
| 903 |
+
|
| 904 |
+
Each line identifies the **center frame** for that sample. The dataloader uses context offsets (e.g., `(-1, 0, +1)`) to also load the temporally adjacent frames.
|
| 905 |
+
|
| 906 |
+
### Minimal Dependencies (no SDK)
|
| 907 |
+
|
| 908 |
+
To build your own loader, you only need:
|
| 909 |
+
```
|
| 910 |
+
numpy
|
| 911 |
+
opencv-python
|
| 912 |
+
laspy==2.6.1
|
| 913 |
+
lazrs==0.8.0
|
| 914 |
+
scipy (for quaternion SLERP — or implement your own)
|
| 915 |
+
```
|
| 916 |
+
|