README.md

---
tags:
- ml-intern
---
# UniDepth Inference: Image → Metric Depth → Point Cloud

A clean, self-contained Python wrapper around **UniDepth** (CVPR 2024 / V2 2025) for converting a single RGB image into a metric 3D point cloud — with **no camera intrinsics required**.

> **Paper**: [UniDepth: Universal Monocular Metric Depth Estimation](https://arxiv.org/abs/2403.18913) (CVPR 2024)  
> **V2 Paper**: [UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler](https://arxiv.org/abs/2502.20110)  
> **Original Code**: [github.com/lpiccinelli-eth/UniDepth](https://github.com/lpiccinelli-eth/UniDepth)  
> **Pretrained Weights**: [`lpiccinelli/unidepth-v2-vits14`](https://huggingface.co/lpiccinelli/unidepth-v2-vits14) · [`lpiccinelli/unidepth-v2-vitl14`](https://huggingface.co/lpiccinelli/unidepth-v2-vitl14)

---

## Installation

You need the original `unidepth` package (which provides the model definitions) plus this wrapper:

```bash
# 1. Install the official UniDepth package from GitHub
pip install git+https://github.com/lpiccinelli-eth/UniDepth.git

# 2. Install this wrapper (or just copy the files into your project)
pip install .
```

**Core dependencies**: `torch`, `torchvision`, `PIL`, `numpy`, `huggingface_hub`, `safetensors`

---

## Quick Start

```python
from PIL import Image
from unidepth.inference import UniDepth

# Load image
image = Image.open("room.jpg").convert("RGB")

# Load model (auto-downloads weights from HuggingFace)
model = UniDepth.from_pretrained("lpiccinelli/unidepth-v2-vits14", device="cuda")

# Run inference → get depth + point cloud
results = model(image)

# Metric depth map [H, W] in meters
depth = results["depth"]

# 3D point cloud [N, 3] in meters (only valid depth pixels)
points = results["points"]

# Predicted camera intrinsics K [3, 3]
intrinsics = results["intrinsics"]

# Save as PLY
from unidepth.inference import save_pointcloud_ply
save_pointcloud_ply("room.ply", points, colors=results["colors"])
```

Command-line:

```bash
python examples/image_to_pointcloud.py room.jpg --output room.ply --checkpoint lpiccinelli/unidepth-v2-vits14
```

---

## How It Works Internally

UniDepth is a **universal monocular metric depth estimator**. Unlike most depth models that only output relative depth or require known camera intrinsics, UniDepth simultaneously predicts:

1. **Camera intrinsics** (self-promptable camera module)
2. **Metric depth** (in meters)
3. **3D ray directions** for every pixel

### 1. Pseudo-Spherical Output Representation

The key innovation is the **pseudo-spherical representation** `(θ, φ, z_log)` instead of the standard Cartesian `(x, y, z)`:

- **θ (azimuth)** — horizontal angle of the camera ray
- **φ (elevation)** — vertical angle of the camera ray  
- **z_log** — log-depth (metric)

Why? Because `(x, y)` in Cartesian backprojection entangles camera rays with depth:
```
x = (u - cx) / fx * z   ← both camera AND depth
y = (v - cy) / fy * z   ← both camera AND depth
```

By predicting angles `(θ, φ)` separately from `z_log`, the model naturally **disentangles camera calibration from depth estimation**. The two tasks don't interfere during training.

### 2. Self-Promptable Camera Module

The camera module bootstraps intrinsics from the image itself:

- Takes the ViT class tokens as initialization
- Runs 2 self-attention layers → predicts 4 scalars `(Δfx, Δfy, Δcx, Δcy)`
- Converts to absolute intrinsics (invariant to image size):
  ```
  fx = Δfx * W / 2
  fy = Δfy * H / 2
  cx = Δcx * W / 2
  cy = Δcy * H / 2
  ```
- Backprojects every pixel through `K⁻¹` to get rays on the unit sphere
- Extracts azimuth/ elevation from those rays → dense camera representation `C`

This means **you don't need to know your camera's focal length** — the model guesses it from visual cues (perspective lines, known object sizes, etc.).

### 3. Depth Module

- Encoder features from DINOv2 ViT at 4 scales (H/14 × W/14 resolution)
- Each scale is **cross-attention conditioned** on the camera embeddings `E = SHE(C)`
  - `SHE` = Spherical Harmonic Encoding (128 channels from 64 harmonics per angle)
- FPN-style decoder with transposed-convolution upsampling
- Final output: `Z_log` upsampled to full `(H, W)` + 2 conv layers

### 4. Converting to Cartesian Point Cloud

The model outputs `O = [θ, φ, Z]` where `Z = exp(Z_log)`. To get standard `(X, Y, Z)`:

```python
X = Z * cos(φ) * cos(θ)
Y = Z * cos(φ) * sin(θ)
Z = Z * sin(φ)
```

This is what `generate_pointcloud()` does for you automatically.

### 5. Handling Known Intrinsics (Optional)

If you already know your camera matrix `K`, you can bypass the predicted camera and use your own:

```python
results = model(image, intrinsics=your_K_matrix)
```

This gives more accurate 3D reconstruction when intrinsics are reliable.

---

## API Reference

### `UniDepth.from_pretrained(checkpoint, device="cuda")`

Load a pretrained model from HuggingFace Hub.

| Checkpoint | Size | Speed | Accuracy |
|-----------|------|-------|----------|
| `lpiccinelli/unidepth-v2-vits14` | 261 MB | Fast | Very Good |
| `lpiccinelli/unidepth-v2-vitl14` | 2.6 GB | Slower | Best |

### `model(image, intrinsics=None) → dict`

Run inference on a PIL Image or tensor.

Returns a dictionary with keys:
- `"depth"` — `[H, W]` metric depth (meters)
- `"confidence"` — `[H, W]` uncertainty (lower = more confident)
- `"points"` — `[N, 3]` Cartesian point cloud (valid pixels only)
- `"colors"` — `[N, 3]` RGB colors for each point
- `"intrinsics"` — `[3, 3]` predicted camera matrix K
- `"camera"` — `[H, W, 2]` predicted azimuth/elevation

### `generate_pointcloud(depth, camera, colors=None, mask=None, intrinsics=None)`

Convert raw model outputs to a filtered point cloud.

### `save_pointcloud_ply(path, points, colors=None)`

Save points (and optional colors) as an ASCII PLY file.

---

## Model Variants

| | UniDepth V1 (CVPR 2024) | UniDepth V2 (2025) |
|---|---|---|
| Backbone | DINOv2 ViT-L | DINOv2 ViT-S / B / L |
| Camera encoding | Spherical Harmonics (81 coefficients) | Sine encoding (64 harmonics) |
| Output | (θ, φ, z_log) | (θ, φ, z_log) |
| Training data | 8 datasets | 23 datasets (16M images) |
| Losses | λ-MSE + Geometric Invariance | + Edge-Guided SSI + Confidence |

This wrapper works with **both V1 and V2** checkpoints.

---

## Notes

- **Dynamic resolution**: The model is trained on variable resolutions (0.2–0.6 MP). You can feed any image size; larger images give finer detail but cost more VRAM.
- **Normalization**: ImageNet normalization `(0.485, 0.456, 0.406)` mean, `(0.229, 0.224, 0.225)` std is applied automatically.
- **Zero-shot**: The model generalizes across indoor, outdoor, and challenging domains without fine-tuning.

---

## Citation

```bibtex
@inproceedings{piccinelli2024unidepth,
  title={UniDepth: Universal Monocular Metric Depth Estimation},
  author={Piccinelli, Luigi and Yang, Yuedong and Sakaridis, Christos and Segu, Mattia and Li, Siyuan and Van Gool, Luc and Yu, Fisher},
  booktitle={CVPR},
  year={2024}
}

@article{piccinelli2025unidepthv2,
  title={UniDepthV2: Universal Monocular Metric Depth Estimation Made Simpler},
  author={Piccinelli, Luigi and Sakaridis, Christos and Yang, Yuedong and Segu, Mattia and Li, Siyuan and Abbeloos, Marc and Van Gool, Luc},
  journal={arXiv:2502.20110},
  year={2025}
}
```

## License

MIT (same as the original repository).

<!-- ml-intern-provenance -->
## Generated by ML Intern

This model repository was generated by [ML Intern](https://github.com/huggingface/ml-intern), an agent for machine learning research and development on the Hugging Face Hub.

- Try ML Intern: https://smolagents-ml-intern.hf.space
- Source code: https://github.com/huggingface/ml-intern

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bdck/unidepth-inference"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)
```

For non-causal architectures, replace `AutoModelForCausalLM` with the appropriate `AutoModel` class.