gassercariad commited on
Commit
317c195
·
verified ·
1 Parent(s): 4680c9a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +824 -33
README.md CHANGED
@@ -20,11 +20,10 @@ thumbnail: >-
20
  https://cdn-uploads.huggingface.co/production/uploads/68d15cb11be9adcdaf269758/lTBcnBZVHuqiI2uxDci5D.jpeg
21
  ---
22
 
23
-
24
  # CARD – Cariad Road Dataset
25
 
26
- > **This page is the central index for all CARD dataset releases.**
27
- > Each sub-dataset has its own repository linked below. The official dataset website (http://card.content.cariad.digital/)
28
 
29
  ---
30
 
@@ -32,17 +31,18 @@ thumbnail: >-
32
 
33
  CARD is a multi-modal driving dataset designed for research in
34
  **depth estimation**, **3D reconstruction**, **object detection**, and related
35
- autonomous-driving tasks. Data was recorded across Germany & Italy and
36
- conditions using a calibrated stereo camera rig paired with 2x LiDAR
37
  and IMU, yielding rich synchronized multi-modal recordings.
38
 
39
  Key properties:
40
- - 🎥 **Stereo camera** images (cam_0 / cam_1) at full resolution
41
  - 📡 **LiDAR** point clouds + IMU signals
42
  - 🏷️ **YOLO-format** bounding-box annotations
43
- - 📐 **Aggregated depth** point clouds (`agg_depth/`) for dense ground-truth depth, which are quasi-dense 3D ground-truth, processed by aggregation of the lidar, while removing dynamic objects artifacts
44
- - 🕐 **Temporal consistency** – all modalities share synchronized timestamps
45
- - 🔒 **Privacy-preserved** – faces and license plates anonymized
 
46
 
47
  ---
48
 
@@ -56,31 +56,47 @@ Key properties:
56
  | [CARD-Italy](https://huggingface.co/datasets/CARD-DATA/CARD-Italy) | Italy | 38 | CC BY-NC 4.0 | Non-commercial only |
57
 
58
  > **Note:** Sequences prefixed with `unused_` in any sub-dataset are not part of
59
- > any official train / val / test split and are provided for zero-shot testing purposes,as an example some sequences which has "slam" in their name often involve multiple loops, which we think will be helpful in SLAM evaluations.
 
60
 
61
  ---
62
 
63
- ## Data Format
64
 
65
  Every sequence across all sub-datasets follows the same folder structure:
66
 
67
  ```
68
  <dataset>/<sequence>/
69
  ├── img/
70
- │ ├── cam_0/ # Left camera images
71
- │ └── cam_1/ # Right camera images
72
- ├── raw/ # LiDAR point clouds + IMU signals
 
73
  ├── labels/ # YOLO-format bounding-box annotations
74
- ├── export/ # Trajectory, calibration, and metadata
75
- └── agg_depth/ # Aggregated depth point clouds (dense GT depth)
76
  ```
77
 
 
 
 
 
78
  ---
79
 
80
  ## Splits
81
 
82
  Official **train / val / test** splits are provided as part of the
83
- [CARD-SDK](https://github.com/cariad-tech/CARD-SDK development kit.
 
 
 
 
 
 
 
 
 
 
 
84
 
85
  ---
86
 
@@ -91,35 +107,810 @@ Official **train / val / test** splits are provided as part of the
91
  | CARD-Germany-Batch1 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
92
  | CARD-Germany-Batch2 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
93
  | CARD-Germany-Batch3 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
94
- | CARD-Italy | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) – **non-commercial only** |
95
 
96
  We have taken reasonable measures to remove personally identifiable information
97
  (e.g., faces and license plates). To request removal of specific images from the
98
  dataset, please contact **gasser.elazab@cariad.technology**.
99
 
100
- The purpose of the CARD project is to help improve road safety and make driving
101
- safer. We encourage use of this dataset toward that goal, and it is forbidden to
102
- use it for any military use.
103
-
104
- ---
105
-
106
- ## Development Kit
107
-
108
- A development kit (CARD-SDK) with tools to load, visualize, and evaluate on the
109
- CARD datasets is going to be released soon.
110
 
111
  ---
112
 
113
  ## Citation
114
 
115
- If you use any CARD sub-dataset in your work, please cite the corresponding entry:
116
 
117
  ```bibtex
118
  @inproceedings{elazab2025card,
119
- title={CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction in Challenging Road Topography},
120
- author={Elazab, Gasser and Neuhaus, Frank and Ko{ss}, Tilman and Splietker, Malte and Date, Aditya and Unterreiner, Michael and Jansen, Maximilian and Hellwich, Olaf},
121
- booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
 
 
 
 
122
  year={2026}
123
  }
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
124
  }
125
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
20
  https://cdn-uploads.huggingface.co/production/uploads/68d15cb11be9adcdaf269758/lTBcnBZVHuqiI2uxDci5D.jpeg
21
  ---
22
 
 
23
  # CARD – Cariad Road Dataset
24
 
25
+ > **This page is the central index for all CARD dataset releases.**
26
+ > Each sub-dataset has its own repository linked below. The official dataset website: http://card.content.cariad.digital/
27
 
28
  ---
29
 
 
31
 
32
  CARD is a multi-modal driving dataset designed for research in
33
  **depth estimation**, **3D reconstruction**, **object detection**, and related
34
+ autonomous-driving tasks. Data was recorded across Germany & Italy
35
+ using a calibrated stereo camera rig paired with 2× LiDAR
36
  and IMU, yielding rich synchronized multi-modal recordings.
37
 
38
  Key properties:
39
+ - 🎥 **Stereo camera** images (`cam_0` / `cam_1`) at full resolution
40
  - 📡 **LiDAR** point clouds + IMU signals
41
  - 🏷️ **YOLO-format** bounding-box annotations
42
+ - 📐 **Aggregated depth** point clouds (`agg_depth/`) — quasi-dense 3D ground truth, processed by aggregation of the LiDAR while removing dynamic-object artifacts
43
+ - 🕐 **Temporal consistency** — all modalities share synchronized microsecond timestamps
44
+ - 🔒 **Privacy-preserved** — faces and license plates anonymized
45
+ - ⛰️ **Challenging topography** — speed bumps, potholes, off-road conditions
46
 
47
  ---
48
 
 
56
  | [CARD-Italy](https://huggingface.co/datasets/CARD-DATA/CARD-Italy) | Italy | 38 | CC BY-NC 4.0 | Non-commercial only |
57
 
58
  > **Note:** Sequences prefixed with `unused_` in any sub-dataset are not part of
59
+ > any official train / val / test split and are provided for zero-shot testing purposes.
60
+ > Sequences with "slam" in their name often involve multiple loops, useful for SLAM evaluations.
61
 
62
  ---
63
 
64
+ ## Data Format Overview
65
 
66
  Every sequence across all sub-datasets follows the same folder structure:
67
 
68
  ```
69
  <dataset>/<sequence>/
70
  ├── img/
71
+ │ ├── cam_0/ # Right camera images
72
+ │ └── cam_1/ # Left camera images
73
+ ├── export/ # Trajectory, calibration, LiDAR, and metadata
74
+ ├── agg_depth/ # Aggregated depth point clouds (dense GT depth)
75
  ├── labels/ # YOLO-format bounding-box annotations
76
+ └── single_depth/ # (optional) single-scan depth
 
77
  ```
78
 
79
+ 👉 **For the complete data format specification** — including every JSON schema, coordinate convention, code recipe, and transformation needed to load the data — see the **[Full Data Format Reference](#full-data-format-reference)** below.
80
+
81
+ > **Tip:** You can copy the full reference section below and paste it into any LLM to generate a working data loader, visualization script, or evaluation pipeline from scratch — no SDK required.
82
+
83
  ---
84
 
85
  ## Splits
86
 
87
  Official **train / val / test** splits are provided as part of the
88
+ [CARD-SDK](https://github.com/cariad-tech/CARD-SDK) development kit.
89
+
90
+ ---
91
+
92
+ ## Development Kit
93
+
94
+ The [CARD-SDK](https://github.com/cariad-tech/CARD-SDK) provides:
95
+ - A PyTorch dataloader with synchronized multi-modal loading
96
+ - A Rerun-based interactive 3D visualizer (`card-viz`)
97
+ - Download utilities for Hugging Face
98
+
99
+ The SDK is **not required** to use the dataset — the full data format reference below is sufficient to write your own loader.
100
 
101
  ---
102
 
 
107
  | CARD-Germany-Batch1 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
108
  | CARD-Germany-Batch2 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
109
  | CARD-Germany-Batch3 | [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
110
+ | CARD-Italy | [CC BY-NC 4.0](https://creativecommons.org/licenses/by-nc/4.0/) — **non-commercial only** |
111
 
112
  We have taken reasonable measures to remove personally identifiable information
113
  (e.g., faces and license plates). To request removal of specific images from the
114
  dataset, please contact **gasser.elazab@cariad.technology**.
115
 
116
+ The purpose of the CARD project is to help improve road safety. It is forbidden to
117
+ use this dataset for any military use.
 
 
 
 
 
 
 
 
118
 
119
  ---
120
 
121
  ## Citation
122
 
123
+ If you use any CARD sub-dataset in your work, please cite:
124
 
125
  ```bibtex
126
  @inproceedings{elazab2025card,
127
+ title={CARD: A Multi-Modal Automotive Dataset for Dense 3D Reconstruction
128
+ in Challenging Road Topography},
129
+ author={Elazab, Gasser and Neuhaus, Frank and Ko{ss}, Tilman and
130
+ Splietker, Malte and Date, Aditya and Unterreiner, Michael and
131
+ Jansen, Maximilian and Hellwich, Olaf},
132
+ booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision
133
+ and Pattern Recognition (CVPR)},
134
  year={2026}
135
  }
136
+ ```
137
+
138
+ ---
139
+ ---
140
+
141
+ # Full Data Format Reference
142
+
143
+ > **Self-contained specification.** Everything below describes the exact file formats, coordinate systems, JSON schemas, and code needed to load and use CARD data. Copy this entire section into any LLM to generate custom tooling.
144
+
145
+ ### Sequence Folder Structure
146
+
147
+ Every sequence across all sub-datasets follows this exact layout:
148
+
149
+ ```
150
+ <data_root>/<sequence_name>/
151
+ ├── img/
152
+ │ ├── cam_0/ # Right camera images
153
+ │ │ ├── cam_0_<timestamp_us>.jpg
154
+ │ │ ├── cam_0_<timestamp_us>.jpg
155
+ │ │ └── ...
156
+ │ └── cam_1/ # Left camera images
157
+ │ ├── cam_1_<timestamp_us>.jpg
158
+ │ └── ...
159
+ ├── export/
160
+ │ ├── output.laz # LiDAR point cloud (compressed LAZ)
161
+ │ ├── output.laz.index.json # Time-indexed access into the LAZ file
162
+ │ ├── output.laz.trajectory.json # Trajectory poses + camera calibration + sensor extrinsics
163
+ │ ├── output.laz.sensor_export.json # IMU + GPS raw data
164
+ │ └── wheel_excitement.json # (optional) wheel excitement analysis
165
+ ├── agg_depth/ # Aggregated depth (dense GT)
166
+ │ ├── cam_0/
167
+ │ │ ├── cam_0_<timestamp_us>.npz
168
+ │ │ └── ...
169
+ │ └── cam_1/
170
+ │ ├── cam_1_<timestamp_us>.npz
171
+ │ └── ...
172
+ ├── labels/ # YOLO-format bounding-box annotations
173
+ │ ├── cam_0_<timestamp_us>.txt
174
+ │ ├── cam_1_<timestamp_us>.txt
175
+ │ └── ...
176
+ └── single_depth/ # (optional) single-scan depth
177
+ ├── cam_0/
178
+ └── cam_1/
179
+ ```
180
+
181
+ **Key naming convention:**
182
+
183
+ All timestamps in filenames are in **microseconds** (µs) as integers. For example:
184
+ `cam_0_1697542123456789.jpg` → timestamp = `1697542123456789` µs.
185
+
186
+ ### Coordinate Systems & Conventions
187
+
188
+ #### Frame Hierarchy
189
+
190
+ ```
191
+ World (inertial, right-handed, Z-up)
192
+ └── Trajectory / Vehicle / Rig Frame (moves with vehicle)
193
+ ├── cam_0 (right camera)
194
+ ├── cam_1 (left camera)
195
+ ├── cariad_wheel_FL_ground
196
+ ├── cariad_wheel_FR_ground
197
+ ├── cariad_wheel_RL_ground
198
+ └── cariad_wheel_RR_ground
199
+ ```
200
+
201
+ #### Axis Conventions
202
+
203
+ | Frame | X | Y | Z | Handedness |
204
+ |---|---|---|---|---|
205
+ | World | East-ish | North-ish | **Up** | Right-handed |
206
+ | Camera | **Right** | **Down** | **Forward** | Right-handed (OpenCV) |
207
+ | Image | **Right** (u) | **Down** (v) | — | Origin top-left |
208
+
209
+ #### Pose Format (7-element vector)
210
+
211
+ All poses in the dataset are stored as:
212
+ ```
213
+ [x, y, z, qw, qx, qy, qz]
214
+ ```
215
+ - `x, y, z` — translation in **meters**
216
+ - `qw, qx, qy, qz` — unit quaternion, **scalar-first** (Hamilton convention)
217
+
218
+ #### Quaternion ↔ Rotation Matrix
219
+
220
+ ```python
221
+ from scipy.spatial.transform import Rotation
222
+ # Given pose = [x, y, z, qw, qx, qy, qz]
223
+ # scipy expects [qx, qy, qz, qw] (scalar-last)
224
+ R = Rotation.from_quat([qx, qy, qz, qw]).as_matrix() # 3×3
225
+ ```
226
+
227
+ #### SE(3) Homogeneous Matrix
228
+
229
+ ```python
230
+ def pose_to_matrix(x, y, z, qw, qx, qy, qz):
231
+ R = Rotation.from_quat([qx, qy, qz, qw]).as_matrix()
232
+ T = np.eye(4)
233
+ T[:3, :3] = R
234
+ T[:3, 3] = [x, y, z]
235
+ return T # 4×4
236
+ ```
237
+
238
+ #### Transform Chain: Camera → World
239
+
240
+ ```python
241
+ # 1. Get vehicle pose in world at time t (interpolated from trajectory)
242
+ T_rig_to_world = pose_to_matrix(*interpolate_pose(traj_data, timestamp_us))
243
+
244
+ # 2. Get camera pose in vehicle frame (static, from calibration)
245
+ T_cam_to_rig = pose_to_matrix(*traj_data["sensor_to_trajectory_poses"]["cam_0"])
246
+
247
+ # 3. Chain them
248
+ T_cam_to_world = T_rig_to_world @ T_cam_to_rig
249
+
250
+ # 4. Transform a 3D point from camera coords to world coords:
251
+ pt_world = T_cam_to_world @ [x_cam, y_cam, z_cam, 1.0]
252
+
253
+ # 5. Transform world points to camera coords (for projection):
254
+ T_world_to_cam = np.linalg.inv(T_cam_to_world)
255
+ pt_cam = T_world_to_cam @ [x_world, y_world, z_world, 1.0]
256
+ ```
257
+
258
+ **Important:** When looking up the trajectory pose for a camera frame, you must add the camera's `timestamp_offset` to the image timestamp:
259
+
260
+ ```python
261
+ trajectory_time = image_timestamp_us + camera_infos[cam_name]["timestamp_offset"]
262
+ pose = interpolate_pose(traj_data, trajectory_time)
263
+ ```
264
+
265
+ ### Trajectory & Calibration JSON (`export/output.laz.trajectory.json`)
266
+
267
+ This is the single most important file in each sequence. It contains the vehicle trajectory, camera intrinsics, distortion parameters, and all sensor extrinsics.
268
+
269
+ ### Full JSON Schema
270
+
271
+ ```jsonc
272
+ {
273
+ "trajectory_poses": [
274
+ // Array of [timestamp_us, pose] pairs, sorted by timestamp ascending
275
+ [1697542100000000, [x, y, z, qw, qx, qy, qz]],
276
+ [1697542100050000, [x, y, z, qw, qx, qy, qz]],
277
+ // ... one entry per trajectory sample (high rate)
278
+ ],
279
+
280
+ "camera_infos": {
281
+ "cam_0": {
282
+ "fx": 1234.56, // focal length X (pixels)
283
+ "fy": 1234.56, // focal length Y (pixels)
284
+ "cx": 960.0, // principal point X (pixels)
285
+ "cy": 540.0, // principal point Y (pixels)
286
+ "image_width": 1920, // image width (pixels)
287
+ "image_height": 1080, // image height (pixels)
288
+ "dist_coeffs": [k1, k2, p1, p2, k3], // OpenCV Brown distortion model (5 params)
289
+ "timestamp_offset": 12345 // offset in microseconds to align camera time to trajectory time
290
+ },
291
+ "cam_1": {
292
+ // same structure
293
+ }
294
+ },
295
+
296
+ "sensor_to_trajectory_poses": {
297
+ // Static transforms: each sensor's pose relative to the vehicle/rig frame
298
+ "cam_0": [x, y, z, qw, qx, qy, qz],
299
+ "cam_1": [x, y, z, qw, qx, qy, qz],
300
+ "cariad_wheel_FL_ground": [x, y, z, qw, qx, qy, qz], // note: capital G variant "Ground" may also appear
301
+ "cariad_wheel_FR_ground": [x, y, z, qw, qx, qy, qz],
302
+ "cariad_wheel_RL_ground": [x, y, z, qw, qx, qy, qz],
303
+ "cariad_wheel_RR_ground": [x, y, z, qw, qx, qy, qz]
304
+ // ... may contain additional sensor frames
305
+ }
306
+ }
307
+ ```
308
+
309
+ ### Camera Intrinsics Matrix (K)
310
+
311
+ ```
312
+ K = | fx 0 cx |
313
+ | 0 fy cy |
314
+ | 0 0 1 |
315
+ ```
316
+
317
+ ### Distortion Model
318
+
319
+ OpenCV Brown's 5-parameter model: `[k1, k2, p1, p2, k3]`
320
+ - `k1, k2, k3` — radial distortion coefficients
321
+ - `p1, p2` — tangential distortion coefficients
322
+
323
+ **Undistortion:**
324
+ ```python
325
+ import cv2
326
+ K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
327
+ dist = np.array(camera_info["dist_coeffs"])
328
+
329
+ # Simple approach:
330
+ undistorted = cv2.undistort(image, K, dist)
331
+
332
+ # Better approach (computes optimal new intrinsics):
333
+ newK, roi = cv2.getOptimalNewCameraMatrix(K, dist, (W, H), alpha=0)
334
+ mapx, mapy = cv2.initUndistortRectifyMap(K, dist, None, newK, (W, H), cv2.CV_32FC1)
335
+ undistorted = cv2.remap(image, mapx, mapy, cv2.INTER_LINEAR)
336
+ # Use newK for all subsequent projection operations on undistorted images
337
+ ```
338
+
339
+ ### Pose Interpolation
340
+
341
+ Trajectory poses are sampled at discrete times. To get the pose at an arbitrary timestamp:
342
+
343
+ ```python
344
+ def interpolate_pose(traj_data, timestamp_us):
345
+ poses = traj_data["trajectory_poses"]
346
+ # Find i such that poses[i-1][0] <= timestamp_us < poses[i][0]
347
+ i = 0
348
+ while i < len(poses) and poses[i][0] <= timestamp_us:
349
+ i += 1
350
+
351
+ if i == 0:
352
+ return poses[0][1] # before first pose: clamp
353
+ if i >= len(poses):
354
+ return poses[-1][1] # after last pose: clamp
355
+
356
+ ts0, pose0 = poses[i - 1]
357
+ ts1, pose1 = poses[i]
358
+ t = (timestamp_us - ts0) / (ts1 - ts0) # interpolation factor [0, 1]
359
+
360
+ # Linear interpolation of translation
361
+ trans = (1 - t) * np.array(pose0[:3]) + t * np.array(pose1[:3])
362
+
363
+ # SLERP for rotation (quaternion scalar-first → convert to scipy scalar-last)
364
+ from scipy.spatial.transform import Rotation, Slerp
365
+ q0_xyzw = [pose0[4], pose0[5], pose0[6], pose0[3]] # qx,qy,qz,qw
366
+ q1_xyzw = [pose1[4], pose1[5], pose1[6], pose1[3]]
367
+ rotations = Rotation.from_quat([q0_xyzw, q1_xyzw])
368
+ slerp = Slerp([0.0, 1.0], rotations)
369
+ q_interp_xyzw = slerp([t]).as_quat()[0]
370
+
371
+ # Back to scalar-first
372
+ qw = q_interp_xyzw[3]
373
+ qx, qy, qz = q_interp_xyzw[:3]
374
+ return [trans[0], trans[1], trans[2], qw, qx, qy, qz]
375
+ ```
376
+
377
+ ### Camera Images
378
+
379
+ ### Location & Naming
380
+
381
+ ```
382
+ <sequence>/img/cam_0/cam_0_<timestamp_us>.jpg # right camera
383
+ <sequence>/img/cam_1/cam_1_<timestamp_us>.jpg # left camera
384
+ ```
385
+
386
+ - `cam_1` = **left** camera, `cam_0` = **right** camera (stereo pair)
387
+ - Filename regex: `cam_\d+_(\d+)\.(?:jpg|jpeg|png)` where group 1 is the timestamp in microseconds
388
+ - Images are stored in their **original (distorted)** form — you must undistort using the calibration
389
+
390
+ ### Loading & Undistortion
391
+
392
+ ```python
393
+ import cv2
394
+ import numpy as np
395
+
396
+ # Load image (OpenCV reads as BGR)
397
+ image = cv2.imread("cam_0_1697542123456789.jpg")
398
+
399
+ # Get calibration from trajectory JSON
400
+ cam_info = traj_data["camera_infos"]["cam_0"]
401
+ fx, fy = cam_info["fx"], cam_info["fy"]
402
+ cx, cy = cam_info["cx"], cam_info["cy"]
403
+ K = np.array([[fx, 0, cx], [0, fy, cy], [0, 0, 1]])
404
+ dist = np.array(cam_info["dist_coeffs"])
405
+
406
+ # Undistort
407
+ undistorted = cv2.undistort(image, K, dist)
408
+ ```
409
+
410
+ ### Stereo Pair Convention
411
+
412
+ | Name | Role | Position |
413
+ |---|---|---|
414
+ | `cam_1` | Left camera | Left on vehicle |
415
+ | `cam_0` | Right camera | Right on vehicle |
416
+
417
+ Stereo timestamps are synchronized to within ~2 ms (2000 µs tolerance).
418
+
419
+ ### Stereo Rectification
420
+
421
+ ```python
422
+ # Get relative transform: left → right
423
+ pose_left = traj_data["sensor_to_trajectory_poses"]["cam_1"] # [x,y,z,qw,qx,qy,qz]
424
+ pose_right = traj_data["sensor_to_trajectory_poses"]["cam_0"]
425
+
426
+ T_left = pose_to_matrix(*pose_left)
427
+ T_right = pose_to_matrix(*pose_right)
428
+ T_right_from_left = np.linalg.inv(T_right) @ T_left
429
+ R = T_right_from_left[:3, :3]
430
+ t = T_right_from_left[:3, 3]
431
+
432
+ # Rectify
433
+ image_size = (cam_info["image_width"], cam_info["image_height"])
434
+ R1, R2, P1, P2, Q, _, _ = cv2.stereoRectify(
435
+ K_left, dist_left, K_right, dist_right, image_size, R, t,
436
+ flags=cv2.CALIB_ZERO_DISPARITY
437
+ )
438
+
439
+ # Build rectification maps
440
+ map1_l, map2_l = cv2.initUndistortRectifyMap(K_left, dist_left, R1, P1, image_size, cv2.CV_16SC2)
441
+ map1_r, map2_r = cv2.initUndistortRectifyMap(K_right, dist_right, R2, P2, image_size, cv2.CV_16SC2)
442
+
443
+ # Apply
444
+ rect_left = cv2.remap(img_left, map1_l, map2_l, cv2.INTER_LINEAR)
445
+ rect_right = cv2.remap(img_right, map1_r, map2_r, cv2.INTER_LINEAR)
446
+
447
+ # Baseline in meters:
448
+ baseline_m = np.linalg.norm(t)
449
+ ```
450
+
451
+ ### LiDAR Point Clouds (LAZ)
452
+
453
+ ### Files
454
+
455
+ ```
456
+ <sequence>/export/output.laz # compressed LiDAR data (all scans)
457
+ <sequence>/export/output.laz.index.json # time-based index for random access
458
+ ```
459
+
460
+ ### Dependencies
461
+
462
+ ```bash
463
+ pip install "laspy==2.6.1" "lazrs==0.8.0"
464
+ ```
465
+
466
+ ### Index JSON Schema
467
+
468
+ ```jsonc
469
+ {
470
+ "entries": [
471
+ {
472
+ "end_ts": 1697542100050000, // end timestamp of this chunk (microseconds)
473
+ "point_offset_in_file": 0, // point offset (NOT byte offset) into LAZ file
474
+ "num_pts": 5000, // number of points in this chunk
475
+ "laser_name": "lidar_0", // which laser scanner produced this chunk
476
+ "part_index": 0 // always 0 (multipart not supported)
477
+ },
478
+ // ... entries sorted by end_ts ascending
479
+ ]
480
+ }
481
+ ```
482
+
483
+ ### Reading LiDAR Points
484
+
485
+ ```python
486
+ import laspy
487
+ import json
488
+ import numpy as np
489
+
490
+ # Load the index
491
+ with open("export/output.laz.index.json") as f:
492
+ index = json.load(f)["entries"]
493
+ index.sort(key=lambda e: e["end_ts"])
494
+
495
+ # Read a specific chunk
496
+ with laspy.open("export/output.laz") as las:
497
+ entry = index[42] # pick an entry
498
+ las.seek(entry["point_offset_in_file"])
499
+ points = las.read_points(entry["num_pts"])
500
+
501
+ # Extract coordinates (world frame, meters)
502
+ coords = np.vstack((points.x, points.y, points.z)).T # shape (N, 3)
503
+
504
+ # Optional: intensity (normalized 0–1)
505
+ if hasattr(points, "intensity"):
506
+ intensity = points.intensity / 65535.0
507
+ ```
508
+
509
+ ### Time-Based Query
510
+
511
+ To get all LiDAR points in a time window `[t0, t1]` (in microseconds):
512
+
513
+ ```python
514
+ import bisect
515
+
516
+ end_times = [e["end_ts"] for e in index]
517
+
518
+ # Select entries with end_ts in (t0, t1]
519
+ lo = bisect.bisect_right(end_times, t0)
520
+ hi = bisect.bisect_right(end_times, t1)
521
+ selected_entries = index[lo:hi]
522
+
523
+ # Read all selected chunks
524
+ all_points = []
525
+ with laspy.open("export/output.laz") as las:
526
+ for entry in selected_entries:
527
+ las.seek(entry["point_offset_in_file"])
528
+ pts = las.read_points(entry["num_pts"])
529
+ all_points.append(np.vstack((pts.x, pts.y, pts.z)).T)
530
+
531
+ points_world = np.concatenate(all_points, axis=0) # shape (N, 3) in world coords
532
+ ```
533
+
534
+ ### LiDAR Coordinate Frame
535
+
536
+ LiDAR points in the LAZ file are in **world coordinates** (same frame as trajectory poses). To project them into a camera:
537
+
538
+ ```python
539
+ # 1. Get camera pose at time t
540
+ T_cam_to_world = T_rig_to_world @ T_cam_to_rig # see "Transform Chain" above
541
+ T_world_to_cam = np.linalg.inv(T_cam_to_world)
542
+
543
+ # 2. Transform points
544
+ pts_homogeneous = np.c_[points_world, np.ones(len(points_world))]
545
+ pts_cam = (T_world_to_cam @ pts_homogeneous.T).T[:, :3]
546
+
547
+ # 3. Project to image (only points in front of camera)
548
+ z = pts_cam[:, 2]
549
+ valid = z > 0
550
+ x, y, z = pts_cam[valid, 0], pts_cam[valid, 1], pts_cam[valid, 2]
551
+ u = (fx * x / z) + cx
552
+ v = (fy * y / z) + cy
553
+
554
+ # 4. Filter to image bounds
555
+ in_bounds = (u >= 0) & (u < W) & (v >= 0) & (v < H)
556
+ ```
557
+
558
+ ### Aggregated Depth (Dense Ground Truth)
559
+
560
+ Aggregated depth files contain quasi-dense 3D point clouds produced by accumulating many LiDAR scans over time and removing dynamic-object artifacts. This is the primary depth ground truth.
561
+
562
+ ### Location & Naming
563
+
564
+ ```
565
+ <sequence>/agg_depth/cam_0/cam_0_<timestamp_us>.npz
566
+ <sequence>/agg_depth/cam_1/cam_1_<timestamp_us>.npz
567
+ ```
568
+
569
+ Each `.npz` file corresponds to one camera frame (matched by timestamp).
570
+
571
+ ### NPZ Contents
572
+
573
+ | Key | Shape | Type | Description |
574
+ |---|---|---|---|
575
+ | `pts_cam` | `(N, 3)` | `float32` | 3D points in **camera frame** coordinates |
576
+
577
+ Camera frame coordinates: X = right, Y = down, Z = forward.
578
+
579
+ ### Loading & Projecting to Depth Map
580
+
581
+ ```python
582
+ import numpy as np
583
+
584
+ data = np.load("agg_depth/cam_0/cam_0_1697542123456789.npz")
585
+ pts_cam = data["pts_cam"].astype(np.float32) # (N, 3) in camera coords
586
+
587
+ # Project to depth map
588
+ H, W = camera_info["image_height"], camera_info["image_width"]
589
+ depth_map = np.zeros((H, W), dtype=np.float32)
590
+
591
+ z = pts_cam[:, 2]
592
+ valid = z > 0 # keep points in front of camera
593
+ x, y, z = pts_cam[valid, 0], pts_cam[valid, 1], pts_cam[valid, 2]
594
+
595
+ u = (fx * x / z) + cx # project to pixel coords
596
+ v = (fy * y / z) + cy
597
+
598
+ in_bounds = (u >= 0) & (u < W) & (v >= 0) & (v < H)
599
+ u, v, z = u[in_bounds], v[in_bounds], z[in_bounds]
600
+
601
+ # Z-buffer: sort far-to-near so near points overwrite far ones
602
+ order = np.argsort(z)[::-1]
603
+ u, v, z = u[order], v[order], z[order]
604
+
605
+ depth_map[v.astype(int), u.astype(int)] = z
606
+ # depth_map is now (H, W) with depth in meters (0 = no data)
607
+ ```
608
+
609
+ ### Transforming to World Coordinates
610
+
611
+ ```python
612
+ # If you need world coordinates from agg_depth pts_cam:
613
+ T_cam_to_rig = pose_to_matrix(*traj_data["sensor_to_trajectory_poses"]["cam_0"])
614
+ pose_rig = interpolate_pose(traj_data, timestamp_us + camera_info["timestamp_offset"])
615
+ T_rig_to_world = pose_to_matrix(*pose_rig)
616
+ T_cam_to_world = T_rig_to_world @ T_cam_to_rig
617
+
618
+ pts_world = (T_cam_to_world @ np.c_[pts_cam, np.ones(len(pts_cam))].T).T[:, :3]
619
+ ```
620
+
621
+ ### Single-Frame Depth
622
+
623
+ Some sequences include single-scan LiDAR depth (one frame, not aggregated).
624
+
625
+ ### Location
626
+
627
+ ```
628
+ <sequence>/single_depth/<cam_name>/<cam_name>_<timestamp_us>.npz
629
+ ```
630
+
631
+ ### NPZ Contents
632
+
633
+ May contain either or both:
634
+
635
+ | Key | Shape | Type | Description |
636
+ |---|---|---|---|
637
+ | `pts_cam` | `(N, 3)` | `float32` | 3D points in camera frame |
638
+ | `depth` | `(H, W)` | `float32` | Pre-projected depth map |
639
+
640
+ Loading logic: use `depth` if present, otherwise project `pts_cam` to a depth map using the same projection formula as aggregated depth.
641
+
642
+ Alternative directory names may be present:
643
+ - `single_lidar/<cam>/`, `lidar_single/<cam>/`, `sparse_lidar/<cam>/`, `lidar_sparse/<cam>/`
644
+
645
+ ### YOLO Bounding-Box Labels
646
+
647
+ ### Location & Naming
648
+
649
+ ```
650
+ <sequence>/labels/<cam_name>_<timestamp_us>.txt
651
+ ```
652
+
653
+ Note: labels are in a flat `labels/` directory (not per-camera subdirectories).
654
+
655
+ ### Format
656
+
657
+ One object per line, space-separated:
658
+ ```
659
+ <class_id> <x_center_norm> <y_center_norm> <width_norm> <height_norm>
660
+ ```
661
+
662
+ All values are **normalized** to `[0, 1]` relative to image dimensions.
663
+
664
+ ### Conversion to Pixel Coordinates
665
+
666
+ ```python
667
+ # Read labels
668
+ H, W = image.shape[:2] # original image dimensions
669
+ with open(label_file) as f:
670
+ for line in f:
671
+ cls, xc, yc, bw, bh = map(float, line.strip().split())
672
+ cls = int(cls)
673
+
674
+ # Convert to pixel coordinates
675
+ x_center_px = xc * W
676
+ y_center_px = yc * H
677
+ width_px = bw * W
678
+ height_px = bh * H
679
+
680
+ # Bounding box corners (top-left, bottom-right)
681
+ x1 = int(x_center_px - width_px / 2)
682
+ y1 = int(y_center_px - height_px / 2)
683
+ x2 = int(x_center_px + width_px / 2)
684
+ y2 = int(y_center_px + height_px / 2)
685
+ ```
686
+
687
+ ### IMU & GPS Data
688
+
689
+ ### File
690
+
691
+ ```
692
+ <sequence>/export/output.laz.sensor_export.json
693
+ ```
694
+
695
+ ### JSON Schema
696
+
697
+ ```jsonc
698
+ {
699
+ "imu": [
700
+ {
701
+ "timestamp": 1697542100000000, // microseconds
702
+ "accel_x": 0.12, // m/s² — X acceleration
703
+ "accel_y": -0.03, // m/s² — Y acceleration
704
+ "accel_z": 9.81, // m/s² — Z acceleration
705
+ "gyro_x": 0.001, // rad/s — X angular velocity
706
+ "gyro_y": -0.002, // rad/s — Y angular velocity
707
+ "gyro_z": 0.0005 // rad/s — Z angular velocity
708
+ },
709
+ // ...
710
+ ],
711
+
712
+ "gps": [
713
+ {
714
+ "timestamp": 1697542100000000, // system timestamp (microseconds)
715
+ "gps_timestamp_us": 1697542100012345, // GPS receiver timestamp (microseconds)
716
+ "lat_rad": 0.8527, // latitude in RADIANS
717
+ "lon_rad": 0.1658 // longitude in RADIANS
718
+ },
719
+ // ...
720
+ ]
721
+ }
722
+ ```
723
+
724
+ ### GPS Conversion
725
+
726
+ ```python
727
+ # Radians to degrees
728
+ lat_deg = gps_entry["lat_rad"] / np.pi * 180.0
729
+ lon_deg = gps_entry["lon_rad"] / np.pi * 180.0
730
+ ```
731
+
732
+ ### IMU Frame
733
+
734
+ IMU measurements are in the vehicle/rig frame (axes aligned with the vehicle body).
735
+
736
+ ### Wheel Excitement Data
737
+
738
+ Optional file present in some sequences.
739
+
740
+ ### File
741
+
742
+ ```
743
+ <sequence>/export/wheel_excitement.json
744
+ ```
745
+
746
+ ### JSON Schema
747
+
748
+ ```jsonc
749
+ {
750
+ "wheel_results": {
751
+ "<frame_id>": { // e.g., "sensor_wheel_FL"
752
+ "wheel_excitement": [
753
+ [timestamp_us, excitement_value], // float pairs
754
+ // ...
755
+ ],
756
+ "contact_point_trajectory": { // (optional)
757
+ "poses": [
758
+ [timestamp_us, [x, y, z, qw, qx, qy, qz]],
759
+ // ...
760
+ ]
761
+ },
762
+ "ground_trajectory": { // (optional, alternative to contact_point_trajectory)
763
+ "poses": [
764
+ [timestamp_us, [x, y, z, qw, qx, qy, qz]],
765
+ // ...
766
+ ]
767
+ }
768
+ },
769
+ // ... one entry per wheel (FL, FR, RL, RR)
770
+ }
771
  }
772
+ ```
773
+
774
+ The wheel identifier (FL, FR, RL, RR) can be extracted from the frame_id by taking the last two characters after splitting by `_`.
775
+
776
+ ### Sensor Extrinsics & Ground Plane
777
+
778
+ ### Sensor Extrinsics
779
+
780
+ All sensor poses relative to the vehicle frame are in `sensor_to_trajectory_poses`:
781
+
782
+ ```python
783
+ # Camera extrinsic (camera → vehicle frame)
784
+ cam_pose = traj_data["sensor_to_trajectory_poses"]["cam_0"] # [x,y,z,qw,qx,qy,qz]
785
+ T_cam_to_rig = pose_to_matrix(*cam_pose)
786
+ ```
787
+
788
+ ### Ground Plane from Wheel Contact Points
789
+
790
+ The four wheel ground contact points define the road plane beneath the vehicle:
791
+
792
+ ```python
793
+ wheel_keys = [
794
+ "cariad_wheel_FL_ground", # or "cariad_wheel_FL_Ground"
795
+ "cariad_wheel_FR_ground",
796
+ "cariad_wheel_RL_ground",
797
+ "cariad_wheel_RR_ground",
798
+ ]
799
+
800
+ # Extract wheel positions (in vehicle frame)
801
+ wheel_points = []
802
+ for key in wheel_keys:
803
+ if key in traj_data["sensor_to_trajectory_poses"]:
804
+ pose = traj_data["sensor_to_trajectory_poses"][key]
805
+ wheel_points.append([pose[0], pose[1], pose[2]])
806
+
807
+ wheel_points = np.array(wheel_points) # (4, 3)
808
+ centroid = np.mean(wheel_points, axis=0)
809
+
810
+ # Fit plane via SVD
811
+ _, _, vh = np.linalg.svd(wheel_points - centroid)
812
+ normal = vh[-1] # plane normal
813
+ if normal[2] < 0:
814
+ normal = -normal # ensure "up" direction
815
+
816
+ # Ground plane: normal · (p - centroid) = 0
817
+ # Typical centroid z ≈ -1.8m (ground is ~1.8m below vehicle center)
818
+ ```
819
+
820
+ ### How To: Core Recipes
821
+
822
+ #### Recipe 1: Load a stereo pair with depth ground truth
823
+
824
+ ```python
825
+ import json, cv2, numpy as np, re, os
826
+
827
+ seq_dir = "/path/to/sequence"
828
+
829
+ # 1. Load calibration
830
+ with open(os.path.join(seq_dir, "export/output.laz.trajectory.json")) as f:
831
+ traj = json.load(f)
832
+
833
+ # 2. Pick a timestamp (e.g., first image)
834
+ cam = "cam_1" # left camera
835
+ cam_dir = os.path.join(seq_dir, "img", cam)
836
+ files = sorted(os.listdir(cam_dir))
837
+ timestamp = int(re.match(r"cam_\d+_(\d+)\.jpg", files[0]).group(1))
838
+
839
+ # 3. Load & undistort image
840
+ ci = traj["camera_infos"][cam]
841
+ K = np.array([[ci["fx"], 0, ci["cx"]], [0, ci["fy"], ci["cy"]], [0, 0, 1]])
842
+ dist = np.array(ci["dist_coeffs"])
843
+ img = cv2.undistort(cv2.imread(os.path.join(cam_dir, files[0])), K, dist)
844
+
845
+ # 4. Load aggregated depth
846
+ npz = np.load(os.path.join(seq_dir, f"agg_depth/{cam}/{cam}_{timestamp}.npz"))
847
+ pts_cam = npz["pts_cam"].astype(np.float32)
848
+
849
+ # 5. Project to depth map
850
+ H, W = ci["image_height"], ci["image_width"]
851
+ # ... (use projection code from "Aggregated Depth" section)
852
+ ```
853
+
854
+ #### Recipe 2: Get LiDAR points for a camera frame
855
+
856
+ ```python
857
+ import bisect
858
+
859
+ # Camera timestamp → trajectory time
860
+ traj_time = timestamp + ci["timestamp_offset"]
861
+
862
+ # Find LAZ entries near this time (e.g., ±50ms window)
863
+ with open(os.path.join(seq_dir, "export/output.laz.index.json")) as f:
864
+ index = json.load(f)["entries"]
865
+ index.sort(key=lambda e: e["end_ts"])
866
+ end_times = [e["end_ts"] for e in index]
867
+
868
+ window_us = 100_000 # 100ms
869
+ t0 = traj_time - window_us // 2
870
+ t1 = traj_time + window_us // 2
871
+ lo = bisect.bisect_right(end_times, t0)
872
+ hi = bisect.bisect_right(end_times, t1)
873
+ entries = index[lo:hi]
874
+
875
+ # Read and project (see LiDAR section for full code)
876
+ ```
877
+
878
+ #### Recipe 3: Relative pose between two frames
879
+
880
+ ```python
881
+ # Useful for multi-view depth estimation, visual odometry, etc.
882
+ T_cam_world_t0 = T_rig_world_t0 @ T_cam_rig
883
+ T_cam_world_t1 = T_rig_world_t1 @ T_cam_rig
884
+
885
+ # Relative pose: t1 expressed in t0's camera frame
886
+ T_t1_in_t0 = np.linalg.inv(T_cam_world_t0) @ T_cam_world_t1
887
+ ```
888
+
889
+ ### Split Files
890
+
891
+ Official **train / val / test** splits are plain text files with one sample per line:
892
+
893
+ ```
894
+ <sequence_name>/img/<cam_name>/<cam_name>_<timestamp_us>.jpg
895
+ ```
896
+
897
+ Example:
898
+ ```
899
+ sequence_001/img/cam_0/cam_0_1697542123456789.jpg
900
+ sequence_001/img/cam_0/cam_0_1697542123556789.jpg
901
+ sequence_002/img/cam_1/cam_1_1697543200000000.jpg
902
+ ```
903
+
904
+ Each line identifies the **center frame** for that sample. The dataloader uses context offsets (e.g., `(-1, 0, +1)`) to also load the temporally adjacent frames.
905
+
906
+ ### Minimal Dependencies (no SDK)
907
+
908
+ To build your own loader, you only need:
909
+ ```
910
+ numpy
911
+ opencv-python
912
+ laspy==2.6.1
913
+ lazrs==0.8.0
914
+ scipy (for quaternion SLERP — or implement your own)
915
+ ```
916
+