Jens-Duttke/Lotus-Depth-D-ONNX

Lotus-D Depth - ONNX

Single-graph ONNX build of Lotus-D depth, a diffusion-based dense prediction model, packaged for stereo and disparity workflows.

Lotus is derived from Stable Diffusion 2.1 but is not a diffusion pipeline at inference: it collapses the formulation to a single training time-step, fixes t = T, and the "-d" (discriminative) variant drops the Gaussian noise input entirely. One deterministic forward pass per image, no scheduler, no ensembling. This build folds the whole chain - VAE encoder, UNet, VAE decoder - into one ONNX file with one input and one output.

The output is a ready-to-use disparity map (near = bright, far = dark) - no postprocessing, no inversion.

See it in action: Oku3D Media Player converts any 2D video or photo into immersive 3D for autostereoscopic and lenticular displays - "Watch everything in 3D."

Key Features

• Deterministic, single-pass: no scheduler loop, no ensembling, no per-frame seed. The same image always yields the same depth map, which matters for video, where a resampled latent would show up as flicker.
• Everything constant is baked in: the empty-prompt CLIP embedding, the depth task embedding, and the timestep are graph initializers. There is no text encoder to ship and no second input to feed.
• Disparity-native: this checkpoint is trained in disparity space, so near = high / far = low comes straight out of the model. No transfer function, no risk of an inverted 3D result.
• Any resolution that is a multiple of 64: height and width are symbolic, and one graph serves 256 through 1024 - verified on both the CPU and DirectML providers.
• FP32 in, FP32 out, FP16 weights: standard boundary types, half-precision storage. The ImageNet to [-1, 1] conversion the Stable Diffusion VAE needs happens inside the graph.
• Opset 21.

Technical Specifications

Property	Value
Input shape	(1, 3, height, width) NCHW, both axes a multiple of 64
Input dtype	float32
Input range	ImageNet-normalized RGB (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225])
Output shape	(1, 1, height, width)
Output dtype	float32
Output range	[0, 5] disparity (higher = closer)
Batch size	fixed at 1
Parameters	951M (UNet 868M, VAE encoder 34M, VAE decoder 50M)
Opset	21

The multiple-of-64 rule is not advisory

The VAE downsamples by 8 and the UNet by a further 8, so the latent grid has to stay divisible by 8. Off-grid sizes fail in two ways, neither of which raises a helpful error:

• A multiple of 8 but not of 64 - for example 518 - returns a smaller map than requested (518 in, 512x512 out). Silently.
• Not a multiple of 8 - for example 500 - fails at runtime inside a Concat in the UNet skip connections.

Sizes verified working: 256, 320, 384, 512, 640, 768, 1024.

Files

File	Weights	Size
lotus-depth-d_fp16_opset21_optimized.onnx	float16	1815 MB

One self-contained file - no external-data companion.

Why there is no 4-bit build. One was made and measured, and it does not earn its place on this architecture. Only 29% of this graph's weight bytes are in MatMul/Gemm; 68% are in convolutions, which MatMulNBits cannot touch. So the file only drops to 1427 MB, while DirectML throughput falls from 36.8 to 2.9 fps at 256x256 and the mean deviation rises to 0.95-2.35% of the output range. The kernel itself is fine in isolation (a quantized 1280x1280 MatMul runs 22% slower than its FP16 equivalent on the same GPU), so the loss comes from how the quantized nodes schedule inside the full graph. On ViT-based depth models the same recipe pays off; here it does not.

Requirements

• ONNX Runtime 1.17 or newer with any execution provider (DirectML, CUDA, CPU).
• Roughly 2 GB of GPU memory at 256x256, rising to 13 GB at 1024x1024 - see Performance.

DirectML: leave the dimensions symbolic

DirectML's graph fusion faults during session creation whenever this graph's input shape is concrete:

RUNTIME_EXCEPTION : Exception during initialization:
  ...\DmlExecutionProvider\src\DmlGraphFusionHelper.cpp ... 80070057

That includes pinning height/width through free-dimension overrides, and it applies to statically exported versions of this graph at any resolution too. Just feed the tensor at the size you want; the graph is fully dynamic. If your host insists on pinning, set the session config entry ep.dml.disable_graph_fusion to 1 - it works, at a cost of roughly a quarter of the throughput.

Quick Start

import cv2
import numpy as np
import onnxruntime as ort

MEAN = np.array([0.485, 0.456, 0.406], dtype=np.float32)
STD = np.array([0.229, 0.224, 0.225], dtype=np.float32)
SIZE = 512  # any multiple of 64

session = ort.InferenceSession(
    "lotus-depth-d_fp16_opset21_optimized.onnx",
    providers=["DmlExecutionProvider", "CPUExecutionProvider"],
)

bgr = cv2.imread("image.jpg")
rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
rgb = cv2.resize(rgb, (SIZE, SIZE), interpolation=cv2.INTER_AREA)
x = (rgb.astype(np.float32) / 255.0 - MEAN) / STD
x = np.transpose(x, (2, 0, 1))[np.newaxis].astype(np.float32)

disparity = session.run(None, {session.get_inputs()[0].name: x})[0]  # (1, 1, S, S)

# near = high; normalize for display
d = disparity.squeeze()
d = (d - d.min()) / max(d.max() - d.min(), 1e-6)
cv2.imwrite("depth.png", (d * 255).astype(np.uint8))

Performance

Benchmarked on an AMD Radeon RX 7900 XTX with ONNX Runtime 1.23 and DirectML, batch size 1, dimensions left symbolic. VRAM is dedicated GPU memory measured on top of the idle baseline.

Resolution	Throughput	Latency	VRAM
256x256	36.8 fps	27 ms	2.0 GB
384x384	22.8 fps	44 ms	2.8 GB
512x512	12.9 fps	78 ms	3.5 GB
768x768	4.4 fps	226 ms	7.0 GB
1024x1024	1.8 fps	547 ms	13.3 GB

Both cost curves are steep: a UNet at 1024x1024 carries 16x the latent tokens of one at 256x256, and attention is quadratic on top of that. 256 and 384 are the video-capable tiers on this class of GPU; 512 and above are for stills unless the GPU is considerably faster.

Accuracy against a full-precision PyTorch reference of the same chain, on the four reference images at 512x512: mean deviation 0.004-0.043% of the output range, rank correlation 0.99997 or better, on both the DirectML and CPU providers.

Comparison: Quality

Side-by-side plasma renders against the strongest variant of each earlier depth-model generation. Plasma convention is uniform (yellow = near, dark = far); every model shown emits this convention natively. The four reference samples are deliberately shared with the sister repos Jens-Duttke/Depth-Anything-3-MONO-ONNX and Jens-Duttke/DepthPro-ONNX-HighPerf so cross-model comparisons are direct.

Model
This Model - Lotus-D (1024)
This Model - Lotus-D (768)
This Model - Lotus-D (512)
This Model - Lotus-D (384)
This Model - Lotus-D (256)
Depth Anything V3 Mono Large (1036)
Depth Anything V3 Mono Large (518)
Depth Anything V2 Large
Depth Anything V2 Small
MiDaS DPT Hybrid

Click any render to view it full size. The comparison models were run at their own native resolutions with their own preprocessing; the reference images are 1:1, which isolates resolution and says nothing about aspect-ratio behaviour.

Three things worth looking for rather than taking on trust:

• Lotus resolves thin structures (wires, twigs, stems) unusually well for its speed class.
• It has no sky-segmentation head, so skies are whatever the model predicts rather than an explicit mask - compare the upper regions against the Depth Anything V3 Mono renders, which do have one.
• At 256 and 384 the VAE decoder leaves a fine-grained noise texture in flat regions, clearly visible in the foliage sample. It fades at 512 and is gone by 768. If your downstream step amplifies local gradients, this is the reason to prefer 512 over 384 even where the frame rate would allow the smaller tier.

Differences from the upstream pipeline

Two deliberate deviations, both measured:

1. The VAE posterior is taken at its mode, not sampled. Upstream calls .latent_dist.sample(), which re-introduces per-frame randomness into a model chosen for determinism. Measured difference across all reference images and resolutions: mean 0.0001-0.0005 of the [0, 1] readout, max 0.034 - below the FP16 noise floor of the rest of the chain.
2. Fixed square input instead of aspect-preserving resize. Upstream scales the longest edge to processing_res and keeps the aspect ratio. This graph takes whatever square you feed it, which is how the consuming application drives every other depth model it ships.

Everything else follows the upstream LotusDPipeline exactly, including the x / 2 + 0.5 denormalization and the mean over the decoder's three output channels.

License

This ONNX build is licensed under the Apache License 2.0; the underlying weights inherit the upstream Lotus license, which is also Apache 2.0. Commercial use, product integration, and service deployment are permitted. This repository is an independent ONNX redistribution and is not affiliated with or endorsed by the upstream Lotus authors.

Acknowledgements

• Lotus by Jing He, Haodong Li, Wei Yin, Yixun Liang, Leheng Li, Kaiqiang Zhou, Hongbo Zhang, Bingbing Liu and Ying-Cong Chen - the upstream model and weights (paper, code).
• Stability AI - Stable Diffusion 2.1, which Lotus is built on.
• The Oku3D Media Player - production consumer of this model.

@article{he2024lotus,
  title={Lotus: Diffusion-based Visual Foundation Model for High-quality Dense Prediction},
  author={He, Jing and Li, Haodong and Yin, Wei and Liang, Yixun and Li, Leheng and Zhou, Kaiqiang and Zhang, Hongbo and Liu, Bingbing and Chen, Ying-Cong},
  journal={arXiv preprint arXiv:2409.18124},
  year={2024}
}