README.md

---
license: bsd-2-clause
library_name: LiteRT
pipeline_tag: depth-estimation
tags:
  - litert
  - tflite
  - on-device
  - android
  - gpu
  - depth-estimation
  - metric-depth
  - metric3d
base_model: yvanyin/metric3d
---

# Metric3D v2 (ViT-S) — LiteRT (on-device, fully-GPU metric depth)

[Metric3D v2](https://github.com/YvanYin/Metric3D) (CVPR/TPAMI 2024) monocular **metric** (absolute,
in-meters) depth, converted to **LiteRT** and running **fully on the `CompiledModel` GPU** (ML Drift) on
Android. Unlike relative-depth models (MiDaS, Depth Anything), Metric3D predicts depth in **meters**. The
DINOv2 ViT-S encoder **and** the RAFT-DPT decoder both ride the GPU delegate — no CPU/ONNX fallback.

![Metric3D v2 — input | metric depth (on-device LiteRT GPU)](samples/sample.png)

## On-device (Pixel 8a, Tensor G3 — verified)

| | |
|---|---|
| nodes on GPU | **2447 / 2447** LITERT_CL (full residency) |
| compile | ~2.2 s (one-time) |
| inference | **~44 ms** (model); ~335 ms full app pipeline |
| size | 78 MB (fp16) |
| accuracy | depth corr **0.96** vs the original Metric3D (0.96–0.98 across indoor 0.7–4 m / mid 4–17 m / outdoor 11–200 m) |

```
image[1,3,448,448] (ImageNet-normalized) →[GPU: DINOv2 ViT-S → RAFT-DPT (4 iters)]→ depth[1,1,448,448] (meters)
```

The model outputs depth for a **canonical camera** (focal 1000 at the canonical resolution). For a
calibrated camera multiply by `fx / 1000` (the de-canonical transform); with no intrinsics the depth is
already in meters and qualitatively correct.

## Preprocessing

Center-crop to square, resize to 448×448, ImageNet normalize in 0–255 scale
`(px − [123.675, 116.28, 103.53]) / [58.395, 57.12, 57.375]`, NCHW planar.

## Usage (Android, LiteRT CompiledModel)

```kotlin
val model = CompiledModel.create(modelPath, CompiledModel.Options(Accelerator.GPU), null)
val input = model.createInputBuffers()
val output = model.createOutputBuffers()
input[0].writeFloat(chw)        // [1,3,448,448] ImageNet-normalized
model.run(input, output)
val depth = output[0].readFloat()   // [448*448] meters
```

A complete Android sample (image picker + depth colormap) is in the official
[google-ai-edge/litert-samples](https://github.com/google-ai-edge/litert-samples) repo under
`compiled_model_api/metric_depth`.

## How it converts (litert-torch)

Fixed 448×448. Encoder = the DINOv2 ViT-S suite (fused-QKV → 4D attention, LayerScale folded into Linear,
baked pos-embed). The RAFT-DPT decoder needs three fixes that **only the on-device run reveals** (desktop
fp16 stays at 0.9999):

1. **Convex upsample → depth-to-space via `ZeroStuffConvT2d`** — the naive "nearest-upsample + in-block
   mask" is exact on desktop but **0.57 on Mali** (`RESIZE_NEAREST` differs at non-stride positions);
   `ZeroStuffConvT2d` masks only stride-aligned positions and the conv kernel supplies the offset.
2. **GELU → accurate tanh approximation** (POW-free); `x·sigmoid(1.702x)` collapses far-depth to **0.51**
   over the 0.1–200 m log-depth bins, tanh restores **0.96**.
3. **`nn.ReLU(inplace=True)` mutates the DPT `ConvBlock` residual** (`relu(x)+convs`) — replicated exactly.

Conversion scripts: in the litert-samples sample's `conversion/` directory.

## License

[BSD-2-Clause](https://github.com/YvanYin/Metric3D/blob/main/LICENSE) (Metric3D); the DINOv2 backbone is
Apache-2.0. Upstream: [YvanYin/Metric3D](https://github.com/YvanYin/Metric3D).