Sensor2Sensor — Camera → LiDAR synthesis checkpoints

Pretrained weights for cross-modal sensor synthesis on nuScenes: given surround camera images, generate the corresponding LiDAR point cloud via a DINOv3-conditioned latent diffusion model over a compact LiDAR range-image VAE.

• Code / docs: https://github.com/skr3178/sensor2sensor
• Trained on: a single RTX 3060 (11.6 GB), nuScenes v1.0-trainval.
• Scope: architecture validation on a small compute budget, not paper-level quality.

Files

File	Model	Params	Size
lidar_vae_best.pt	v5 LiDAR range-image VAE (encoder μ + decoder)	~2.07 M	8.3 MB
lidar_unet_best.pt	850-scenes DINOv3-conditioned diffusion U-Net (best held-out CD)	—	59 MB
lidar_unet_ema.pt	EMA weights of the diffusion U-Net	—	59 MB

Held-out metrics (Chamfer distance, metres, lower = better; cfg=3.5, DDIM-25)

Component	Metric	Value
LiDAR VAE (v5)	CD-VAE-only (decode(μ) vs raw)	0.791 m
Diffusion U-Net (850-scenes)	CD-3D-raw (N=16 held-out)	1.994 m
Diffusion U-Net (850-scenes)	CD-BEV (N=16 held-out)	1.220 m
End-to-end (VAE + diffusion)	CD-3D-raw (4 held-out keyframes)	3.036 m

The 850-scenes checkpoint is selected by held-out Chamfer distance measured in-loop, not by training MSE (a checkpoint with lower training MSE generalized worse). See the repo's s2s_min/RESULTS.md §15 for the full rationale.

Loading

import torch
ckpt = torch.load("lidar_vae_best.pt", map_location="cuda")
# state dict keyed by the training-time best l1_range_ema basin; see the GitHub repo
# (s2s_min/models/) for the matching module definitions.

License

MIT (see the GitHub repository). nuScenes data is subject to its own license/terms.