--- license: apache-2.0 tags: - depth-estimation - normal-estimation - geometry-estimation - flux - comfyui pipeline_tag: depth-estimation --- # Lotus & Lotus-2: Visual Foundation Models for Dense Geometry Estimation This repository provides consolidated, plug-and-play model weights for both **Lotus-2 (FLUX.1-dev SOTA)** and **Lotus-1 (SD 2.1 Fast)** for monocular **depth** and **surface normal** estimation, pre-configured for native integration with [ComfyUI-Lotus-2](https://github.com/1038lab/ComfyUI-Lotus-2). [![Project Website](https://img.shields.io/badge/Lotus--2-Project%20Website-pink?logo=googlechrome&logoColor=white)](https://lotus-2.github.io/) [![arXiv Paper](https://img.shields.io/badge/arXiv-2512.01030-b31b1b?logo=arxiv&logoColor=white)](https://huggingface.co/papers/2512.01030) [![Official GitHub](https://img.shields.io/badge/Official%20Code-EnVision--Research%2FLotus--2-black?logo=github)](https://github.com/EnVision-Research/Lotus-2) [![ComfyUI Node](https://img.shields.io/badge/ComfyUI-1038lab%2FComfyUI--Lotus2-blue?logo=github)](https://github.com/1038lab/ComfyUI-Lotus-2) [![HuggingFace Demo (Depth)](https://img.shields.io/badge/🤗%20HuggingFace-Demo%20(Depth)-yellow)](https://huggingface.co/spaces/haodongli/Lotus-2_Depth) [![HuggingFace Demo (Normal)](https://img.shields.io/badge/🤗%20HuggingFace-Demo%20(Normal)-yellow)](https://huggingface.co/spaces/haodongli/Lotus-2_Normal) ![teaser](assets/teaser-1.jpg) --- ## 📦 Repository Contents All safetensors weights in this repository are verified, standalone, and organized for direct automatic downloading or manual placement. ### 1. Lotus-2 Weights (FLUX.1-dev Monocular Geometry) Developed by EnVision-Research (2026), built on FLUX.1-dev DiT using multi-stage LoRA and Local Continuity Modules (LCM): | File Name | Size | Task | Description | | :--- | :--- | :--- | :--- | | `lotus-2_core_predictor_depth.safetensors` | ~1.43 GB | Depth | LoRA adapter for core global depth prediction | | `lotus-2_detail_sharpener_depth.safetensors` | ~1.43 GB | Depth | LoRA adapter for high-frequency depth refinement | | `lotus-2_lcm_depth.safetensors` | ~39 kB | Depth | Local Continuity Module for depth smoothness | | `lotus-2_core_predictor_normal.safetensors` | ~2.87 GB | Normal | LoRA adapter for core surface normal prediction | | `lotus-2_detail_sharpener_normal.safetensors` | ~2.87 GB | Normal | LoRA adapter for high-frequency normal refinement | | `lotus-2_lcm_normal.safetensors` | ~39 kB | Normal | Local Continuity Module for normal consistency | | `ae.safetensors` | ~335 MB | VAE | FLUX.1-dev native autoencoder | ### 2. Lotus-1 Weights (SD 2.1 Fast UNet Engine) Developed by EnVision-Research, lightweight standalone diffusion UNet models (~1.7 GB): | File Name | Size | Task | Description | | :--- | :--- | :--- | :--- | | `lotus-depth-g-v2-1-disparity-fp16.safetensors` | ~1.74 GB | Depth | Standalone Generative Depth (disparity, FP16) | | `lotus-normal-g-v1-1-fp16.safetensors` | ~1.74 GB | Normal | Standalone Generative Surface Normal (FP16) | | `vae-ft-mse-840000-ema-pruned.safetensors` | ~335 MB | VAE | Stable Diffusion 2.1 fine-tuned MSE Autoencoder | --- ## 🚀 Usage in ComfyUI These models are natively supported by the [ComfyUI-Lotus-2](https://github.com/1038lab/ComfyUI-Lotus-2) custom node suite under category `🧪AILab/Geometry`: ```bash cd ComfyUI/custom_nodes git clone https://github.com/1038lab/ComfyUI-Lotus-2.git ``` ### Node 1: `Lotus 2 (FLUX)` (SOTA Quality) - **Minimal Inputs**: Requires only `image` and `model` (FLUX.1-dev UNet / Checkpoint). - **Zero VRAM Waste**: VAE and text conditioning are handled internally—no need to connect DualCLIPLoader or load 10GB T5 models! - **Auto-Download**: If LoRA or VAE files are not found locally, the node automatically downloads them from this repository (`1038lab/Lotus-2`). ### Node 2: `Lotus` (SD 2.1 Fast Engine) - **Standalone**: All-in-one node without requiring an external base model loader. - **Fast**: Generates high-quality geometry maps in seconds. --- ## 👏 Credits & Acknowledgments We express our sincere gratitude and full credit to the original research teams and creators: ### Lotus-2 (FLUX-based) - **Authors**: [Jing He](https://scholar.google.com/citations?hl=en&user=RsLS11MAAAAJ)1, [Haodong Li](https://haodong-li.com/)1,2*, [Mingzhi Sheng](https://github.com/MingzhiSheng)1*, [Ying-Cong Chen](https://www.yingcong.me/)1,3✉ - **Affiliations**: 1HKUST (GZ), 2HKUST, 3HKUST Dept. of CSE - **Original Code & Weights**: [EnVision-Research/Lotus-2](https://github.com/EnVision-Research/Lotus-2) / [jingheya/Lotus-2](https://huggingface.co/jingheya/Lotus-2) ### Lotus-1 (SD-based) - **Authors**: Jing He, Haodong Li, Weicai Ye, Wang Zhao, Suping Chen, Torsten Sattler, Guofeng Zhang, Shenghua Gao, Ying-Cong Chen - **Original Code & Weights**: [EnVision-Research/Lotus](https://github.com/EnVision-Research/Lotus) / [jingheya/lotus-*](https://huggingface.co/jingheya) ### Base Models - **FLUX.1-dev & FLUX VAE**: [Black Forest Labs](https://blackforestlabs.ai/) - **Stable Diffusion 2.1 & VAE**: [Stability AI](https://stability.ai/) --- ## 📑 Citation If you find Lotus or Lotus-2 helpful in your research or projects, please cite the official papers: ```bibtex @article{he2025lotus2, title = {Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model}, author = {He, Jing and Li, Haodong and Sheng, Mingzhi and Chen, Ying-Cong}, journal = {arXiv preprint arXiv:2512.01030}, year = {2025} } @article{he2024lotus, title = {Lotus: Diffusion-based Visual Foundation Model for Dense Geometry Estimation}, author = {He, Jing and Li, Haodong and Ye, Weicai and Zhao, Wang and Chen, Suping and Sattler, Torsten and Zhang, Guofeng and Gao, Shenghua and Chen, Ying-Cong}, journal = {arXiv preprint arXiv:2409.08272}, year = {2024} } ```