README.md

---
license: mit
tags:
  - lidar
  - diffusion
  - autonomous-driving
  - sensor-synthesis
  - nuscenes
  - dinov3
library_name: pytorch
---

# Sensor2Sensor — Camera → LiDAR synthesis checkpoints

Pretrained weights for **cross-modal sensor synthesis** on nuScenes: given surround
camera images, generate the corresponding LiDAR point cloud via a
**DINOv3-conditioned latent diffusion** model over a compact **LiDAR range-image VAE**.

- **Code / docs:** https://github.com/skr3178/sensor2sensor
- **Trained on:** a single RTX 3060 (11.6 GB), nuScenes v1.0-trainval.
- **Scope:** architecture validation on a small compute budget, not paper-level quality.

## Files

| File | Model | Params | Size |
|---|---|---|---|
| `lidar_vae_best.pt` | v5 LiDAR range-image VAE (encoder μ + decoder) | ~2.07 M | 8.3 MB |
| `lidar_unet_best.pt` | 850-scenes DINOv3-conditioned diffusion U-Net (best held-out CD) | — | 59 MB |
| `lidar_unet_ema.pt`  | EMA weights of the diffusion U-Net | — | 59 MB |

## Held-out metrics (Chamfer distance, metres, lower = better; cfg=3.5, DDIM-25)

| Component | Metric | Value |
|---|---|---|
| LiDAR VAE (v5) | `CD-VAE-only` (decode(μ) vs raw) | **0.791 m** |
| Diffusion U-Net (850-scenes) | `CD-3D-raw` (N=16 held-out) | **1.994 m** |
| Diffusion U-Net (850-scenes) | `CD-BEV` (N=16 held-out) | **1.220 m** |
| End-to-end (VAE + diffusion) | `CD-3D-raw` (4 held-out keyframes) | **3.036 m** |

The 850-scenes checkpoint is selected by **held-out Chamfer distance measured in-loop**,
not by training MSE (a checkpoint with lower training MSE generalized worse). See the
repo's `s2s_min/RESULTS.md` §15 for the full rationale.

## Loading

```python
import torch
ckpt = torch.load("lidar_vae_best.pt", map_location="cuda")
# state dict keyed by the training-time best l1_range_ema basin; see the GitHub repo
# (s2s_min/models/) for the matching module definitions.
```

## License

MIT (see the GitHub repository). nuScenes data is subject to its own license/terms.