heyleandro/DINOcular

Overview

Model weights and inference code for DINOcular - a self-supervised RGB-D encoder that learns joint visuospatial representations from RGB-D observations.

project page , arxiv

Input

• rgb: image tensor of shape (B, 3, H, W), normalized with ImageNet stats
• depth: depth map tensor of shape (B, 1, H, W) (same spatial size as the image), raw metric depth (i.e. in meters)

Requirements

torch
timm

Usage

import torch
from DINOcular import DINOcular_S

chkpt = torch.load("dinocular_s.pth", map_location="cpu", weights_only=False)
model = DINOcular_S()
model.load_state_dict(chkpt, strict=False)
model.eval()

rgb   = torch.randn(1, 3, 224, 224)   # IMAGENET normalized RGB image
depth = torch.randn(1, 1, 224, 224)   # raw metric depth (in meters) 

with torch.no_grad():
    features = model(rgb, depth) # output features