LDCM

LDCM reconstructs dense metric depth from a single RGB image and sparse metric depth observations. The model combines a frozen MoGe monocular geometry prior, multi-scale Poisson completion, and a DINOv2-based refinement network.

The checkpoint in this repository is ldcm.pt.

Installation

conda create -n ldcm python=3.10 -y
conda activate ldcm

pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu126

git clone https://github.com/aigc3d/LDCM.git
cd LDCM

git clone https://github.com/EasternJournalist/utils3d.git
cd utils3d
git checkout 3fab839f0be9931dac7c8488eb0e1600c236e183
pip install .
cd ..

pip install -r requirements.txt
pip install -e .

Quick Start

The bundled demo assets in the GitHub repository can be used directly. Inputs are RGB tensors in [0, 1] and sparse metric depth tensors in meters. Missing sparse-depth pixels should be 0.

from pathlib import Path

import numpy as np
import torch
from PIL import Image

from ldcm import LDCMModel

device = torch.device("cuda")
sample_dir = Path("assets/sample_1")

image_np = np.asarray(Image.open(sample_dir / "image.png").convert("RGB"), dtype=np.float32) / 255.0
image = torch.from_numpy(image_np).permute(2, 0, 1).unsqueeze(0).to(device)

prior_np = np.load(sample_dir / "sparse_depth.npy").astype(np.float32)
prior = torch.from_numpy(prior_np).unsqueeze(0).unsqueeze(0).to(device)

model = LDCMModel.from_pretrained(
    "pkqbajng/LDCM",
    moge_path="Ruicheng/moge-2-vits-normal",
).to(device).eval()

with torch.inference_mode():
    output = model.infer(image, prior)

depth = output["depth_pred"]      # [B, 1, H, W]
points = output["points_pred"]    # [B, H, W, 3]
mask = output["mask"]             # [B, 1, H, W], MoGe valid-region mask

output["mask"] is the MoGe valid-region mask used during completion. It is not a separately trained LDCM prediction mask.

Poisson Completion Usage

For data refinement, you can also expose the MoGe prior and Poisson completion step explicitly.

from pathlib import Path

import numpy as np
import torch
from PIL import Image

from ldcm.moge.model.v2 import MoGeModel
from ldcm.poisson_completion import poisson_completion

device = torch.device("cuda")
sample_dir = Path("assets/sample_1")

image_np = np.asarray(Image.open(sample_dir / "image.png").convert("RGB"), dtype=np.float32) / 255.0
image = torch.from_numpy(image_np).permute(2, 0, 1).unsqueeze(0).to(device)

sparse_np = np.load(sample_dir / "sparse_depth.npy").astype(np.float32)
sparse_depth = torch.from_numpy(sparse_np).unsqueeze(0).unsqueeze(0).to(device)

moge = MoGeModel.from_pretrained("Ruicheng/moge-2-vits-normal").to(device).eval()

with torch.no_grad():
    moge_output = moge.infer(image, apply_mask=False)
    mono_depth = moge_output["depth"].unsqueeze(1)  # [B, 1, H, W]

    completed_depth = poisson_completion(
        sparse=sparse_depth,
        mono_depth=mono_depth,
        num_scales=4,
        thres=3.0,
        lamda=5.0,
        rtol=1e-5,
        max_iter_per_scale=[5000, 2000, 1000, 500],
        max_resolution_ratio=1.0,
    )

The output completed_depth has shape [B, 1, H, W]. The solver first aligns the monocular prior to the sparse metric depth and then runs multi-scale Poisson optimization from coarse to fine.

Links

• LDCM project page: https://pkqbajng.github.io/ldcm/
• Paper: https://arxiv.org/abs/2605.30115
• GitHub repository: https://github.com/aigc3d/LDCM

License

CC BY-NC 4.0.