README.md

metadata

license: apache-2.0
tags:
  - depth-estimation
  - normal-estimation
  - geometry-estimation
  - flux
  - comfyui
pipeline_tag: depth-estimation

Lotus & Lotus-2: Visual Foundation Models for Dense Geometry Estimation

This repository provides consolidated, plug-and-play model weights for both Lotus-2 (FLUX.1-dev SOTA) and Lotus-1 (SD 2.1 Fast) for monocular depth and surface normal estimation, pre-configured for native integration with ComfyUI-Lotus-2.

---

📦 Repository Contents

All safetensors weights in this repository are verified, standalone, and organized for direct automatic downloading or manual placement.

1. Lotus-2 Weights (FLUX.1-dev Monocular Geometry)

Developed by EnVision-Research (2026), built on FLUX.1-dev DiT using multi-stage LoRA and Local Continuity Modules (LCM):

File Name	Size	Task	Description
lotus-2_core_predictor_depth.safetensors	~1.43 GB	Depth	LoRA adapter for core global depth prediction
lotus-2_detail_sharpener_depth.safetensors	~1.43 GB	Depth	LoRA adapter for high-frequency depth refinement
lotus-2_lcm_depth.safetensors	~39 kB	Depth	Local Continuity Module for depth smoothness
lotus-2_core_predictor_normal.safetensors	~2.87 GB	Normal	LoRA adapter for core surface normal prediction
lotus-2_detail_sharpener_normal.safetensors	~2.87 GB	Normal	LoRA adapter for high-frequency normal refinement
lotus-2_lcm_normal.safetensors	~39 kB	Normal	Local Continuity Module for normal consistency
ae.safetensors	~335 MB	VAE	FLUX.1-dev native autoencoder

2. Lotus-1 Weights (SD 2.1 Fast UNet Engine)

Developed by EnVision-Research, lightweight standalone diffusion UNet models (~1.7 GB):

File Name	Size	Task	Description
lotus-depth-g-v2-1-disparity-fp16.safetensors	~1.74 GB	Depth	Standalone Generative Depth (disparity, FP16)
lotus-normal-g-v1-1-fp16.safetensors	~1.74 GB	Normal	Standalone Generative Surface Normal (FP16)
vae-ft-mse-840000-ema-pruned.safetensors	~335 MB	VAE	Stable Diffusion 2.1 fine-tuned MSE Autoencoder

---

🚀 Usage in ComfyUI

These models are natively supported by the ComfyUI-Lotus-2 custom node suite under category 🧪AILab/Geometry:

cd ComfyUI/custom_nodes
git clone https://github.com/1038lab/ComfyUI-Lotus-2.git

Node 1: Lotus 2 (FLUX) (SOTA Quality)

• Minimal Inputs: Requires only image and model (FLUX.1-dev UNet / Checkpoint).
• Zero VRAM Waste: VAE and text conditioning are handled internally—no need to connect DualCLIPLoader or load 10GB T5 models!
• Auto-Download: If LoRA or VAE files are not found locally, the node automatically downloads them from this repository (1038lab/Lotus-2).

Node 2: Lotus (SD 2.1 Fast Engine)

• Standalone: All-in-one node without requiring an external base model loader.
• Fast: Generates high-quality geometry maps in seconds.

---

👏 Credits & Acknowledgments

We express our sincere gratitude and full credit to the original research teams and creators:

Lotus-2 (FLUX-based)

• Authors: Jing He¹, Haodong Li^1,2*, Mingzhi Sheng^1*, Ying-Cong Chen^1,3✉
• Affiliations: ¹HKUST (GZ), ²HKUST, ³HKUST Dept. of CSE
• Original Code & Weights: EnVision-Research/Lotus-2 / jingheya/Lotus-2

Lotus-1 (SD-based)

• Authors: Jing He, Haodong Li, Weicai Ye, Wang Zhao, Suping Chen, Torsten Sattler, Guofeng Zhang, Shenghua Gao, Ying-Cong Chen
• Original Code & Weights: EnVision-Research/Lotus / jingheya/lotus-*

Base Models

• FLUX.1-dev & FLUX VAE: Black Forest Labs
• Stable Diffusion 2.1 & VAE: Stability AI

---

📑 Citation

If you find Lotus or Lotus-2 helpful in your research or projects, please cite the official papers:

@article{he2025lotus2,
  title   = {Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model},
  author  = {He, Jing and Li, Haodong and Sheng, Mingzhi and Chen, Ying-Cong},
  journal = {arXiv preprint arXiv:2512.01030},
  year    = {2025}
}

@article{he2024lotus,
  title   = {Lotus: Diffusion-based Visual Foundation Model for Dense Geometry Estimation},
  author  = {He, Jing and Li, Haodong and Ye, Weicai and Zhao, Wang and Chen, Suping and Sattler, Torsten and Zhang, Guofeng and Gao, Shenghua and Chen, Ying-Cong},
  journal = {arXiv preprint arXiv:2409.08272},
  year    = {2024}
}