metadata
license: mit
tags:
- lidar
- diffusion
- autonomous-driving
- sensor-synthesis
- nuscenes
- dinov3
library_name: pytorch
Sensor2Sensor — Camera → LiDAR synthesis checkpoints
Pretrained weights for cross-modal sensor synthesis on nuScenes: given surround camera images, generate the corresponding LiDAR point cloud via a DINOv3-conditioned latent diffusion model over a compact LiDAR range-image VAE.
- Code / docs: https://github.com/skr3178/sensor2sensor
- Trained on: a single RTX 3060 (11.6 GB), nuScenes v1.0-trainval.
- Scope: architecture validation on a small compute budget, not paper-level quality.
Files
| File | Model | Params | Size |
|---|---|---|---|
lidar_vae_best.pt |
v5 LiDAR range-image VAE (encoder μ + decoder) | ~2.07 M | 8.3 MB |
lidar_unet_best.pt |
850-scenes DINOv3-conditioned diffusion U-Net (best held-out CD) | — | 59 MB |
lidar_unet_ema.pt |
EMA weights of the diffusion U-Net | — | 59 MB |
Held-out metrics (Chamfer distance, metres, lower = better; cfg=3.5, DDIM-25)
| Component | Metric | Value |
|---|---|---|
| LiDAR VAE (v5) | CD-VAE-only (decode(μ) vs raw) |
0.791 m |
| Diffusion U-Net (850-scenes) | CD-3D-raw (N=16 held-out) |
1.994 m |
| Diffusion U-Net (850-scenes) | CD-BEV (N=16 held-out) |
1.220 m |
| End-to-end (VAE + diffusion) | CD-3D-raw (4 held-out keyframes) |
3.036 m |
The 850-scenes checkpoint is selected by held-out Chamfer distance measured in-loop,
not by training MSE (a checkpoint with lower training MSE generalized worse). See the
repo's s2s_min/RESULTS.md §15 for the full rationale.
Loading
import torch
ckpt = torch.load("lidar_vae_best.pt", map_location="cuda")
# state dict keyed by the training-time best l1_range_ema basin; see the GitHub repo
# (s2s_min/models/) for the matching module definitions.
License
MIT (see the GitHub repository). nuScenes data is subject to its own license/terms.