| --- |
| license: mit |
| tags: |
| - depth-estimation |
| - diffusion |
| - monocular-depth |
| - lidar-prompted |
| - pixel-perfect-depth |
| - pytorch |
| --- |
| |
| # LiDAR-Perfect Depth (LPD) |
|
|
| Implementation of *LiDAR-Perfect Depth: Score-Decomposed Diffusion with Kalman-in-the-Loop Denoising for Sparse-Prompted Depth* on top of [Pixel-Perfect Depth](https://github.com/gangweix/pixel-perfect-depth). |
|
|
| ## Repos |
|
|
| | Repo | Contents | Size | |
| |---|---|---| |
| | `chenming-wu/LiDAR-Perfect-Depth` (this) | code + 6 LPD-DA2 checkpoints + inference vis + extraction helper | 12.5 GB | |
| | [`chenming-wu/LiDAR-Perfect-Depth-Datasets`](https://huggingface.co/datasets/chenming-wu/LiDAR-Perfect-Depth-Datasets) | extracted eval sets + training-set archives + PPD/DA-V2/RAFT weights | ~991 GB | |
|
|
| ## What's in this repo |
|
|
| - `code/` — full LPD codebase. New modules under `ppd/lpd/`; updated configs and adapter loaders. |
| - `checkpoints/` |
| - `e000-s001000.ckpt` … `e004-s005000.ckpt` — per-epoch fine-tuned weights (DA2 backbone, 5K steps) |
| - `last.ckpt` — same as e004 |
| - 2.0 GB each, weights-only |
| - `inference_vis/` — 8-sample qualitative panels (RGB | GT | PPD | LPD | LPD-variance) |
| - `extract_archives.sh` — extracts the dataset archives back into the layout the code expects |
|
|
| ## Backbone options |
|
|
| PPD (and therefore LPD) support two semantic-prompt backbones — switching is a single config change. |
|
|
| | Backbone | Trainable params | PPD ckpt | Config | |
| |---|---|---|---| |
| | **DA2** (Depth-Anything-V2 ViT-L) | 16.3 M | [`gangweix/Pixel-Perfect-Depth/ppd.pth`](https://huggingface.co/gangweix/Pixel-Perfect-Depth/resolve/main/ppd.pth) | `code/ppd/configs/lpd_run5d_10k.yaml` | |
| | **MoGe2** | 16.3 M | [`chenming-wu/LiDAR-Perfect-Depth-Datasets/pretrained/ppd_moge2/`](https://huggingface.co/datasets/chenming-wu/LiDAR-Perfect-Depth-Datasets/tree/main/pretrained/ppd_moge2) | `code/ppd/configs/lpd_run5d_moge2.yaml` | |
|
|
| The official PPD MoGe2 release reports 20-30 % improvement over DA2 on zero-shot benchmarks; the LPD prompt branch is identical for either backbone. |
|
|
| ## Training run summary (DA2 backbone, 5K steps) |
|
|
| ``` |
| Backbone: PPD-DA2 — 820 M, frozen |
| Trainable: 16 M (sparse-prompt encoder + gate) |
| Resolution: 1024 × 768 |
| Batch: 18 (~133 GB peak on a single H200) |
| Steps: 5,000 (5 epochs × 1000 batches) |
| Mix: Hypersim 0.5 / UrbanSyn 0.15 / UnrealStereo4K 0.15 / VKITTI2 0.1 / TartanAir 0.1 |
| Init: gangweix/Pixel-Perfect-Depth ppd.pth |
| |
| epoch 0 → 4 loss: 0.0186 → 0.0177 (-4.8%) |
| ``` |
|
|
| The 5 K steps shown here is a partial-train demo — paper-scale training would push it much further. |
|
|
| For MoGe2 backbone we verified the full forward + backward + checkpointing pipeline works |
| (`lpd_run5d_moge2.yaml`); training to convergence is left for a multi-GPU run. |
|
|
| ## Verification |
|
|
| ```bash |
| cd code/ |
| pip install -r requirements.txt |
| python -m ppd.lpd.tests.verify_paper # 30 paper claims, all pass |
| ``` |
|
|
| `PAPER_CHECKLIST.md` maps each section of the paper to specific code files/lines. |
|
|
| ## Reproducing training |
|
|
| ```bash |
| # Pretrained inputs the code expects |
| ln -sf <ppd.pth> code/checkpoints/ppd.pth # DA2 |
| ln -sf <depth_anything_v2_vitl.pth> code/checkpoints/depth_anything_v2_vitl.pth |
| # OR for MoGe2: |
| ln -sf <ppd_moge2.pth> code/checkpoints/ppd_moge2.pth |
| ln -sf <moge2.pt> code/checkpoints/moge2.pt |
| |
| # Hypersim 512² pretrain (DA2) |
| bash code/train_lpd.sh |
| |
| # 5-dataset 1024×768 fine-tune |
| python code/main.py --cfg_file code/ppd/configs/lpd_run5d_10k.yaml # DA2 |
| # OR |
| python code/main.py --cfg_file code/ppd/configs/lpd_run5d_moge2.yaml # MoGe2 |
| |
| # Inference comparison (PPD vs LPD) |
| python code/experiments/eval_lpd_vs_ppd.py |
| ``` |
|
|
| ## Datasets — fetching + extracting |
|
|
| Everything is hosted as **archives** in the dataset repo to keep file counts low and upload bandwidth high. |
|
|
| ```bash |
| hf download chenming-wu/LiDAR-Perfect-Depth-Datasets --repo-type dataset \ |
| --local-dir /mnt/sig/datasets |
| |
| bash code/extract_archives.sh /mnt/sig/datasets /mnt/sig/datasets/archives |
| ``` |
|
|
| Layout after extraction matches what the LPD configs reference (`/mnt/sig/datasets/train/<scene>/...`, `/mnt/sig/datasets/eval_image/...`). |
|
|
| ## Citations |
|
|
| - **Pixel-Perfect Depth** — Xu et al., NeurIPS 2025 |
| - **LiDAR-Perfect Depth** — `paper.tex` in this repo |
|
|
