README.md

---
license: apache-2.0
language:
  - en
tags:
  - depth-estimation
  - colonoscopy
  - medical-imaging
  - video
  - lora
  - diffusion
library_name: transformers
base_model:
  - tencent/DepthCrafter
  - stabilityai/stable-video-diffusion-img2vid-xt
pipeline_tag: depth-estimation
---

# ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors

ColonCrafter builds upon [DepthCrafter](https://huggingface.co/tencent/DepthCrafter) and [Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt) to provide temporally consistent depth predictions for colonoscopy video.

## Model Details

- **Model Type:** Video Depth Estimation (Diffusion-based)
- **Base Architecture:** DepthCrafter UNet with LoRA adaptation
- **LoRA Configuration:**
  - Rank: 16
  - Target modules: `to_q`, `to_k`, `to_v`, `to_out.0`
  - Dropout: 0.1
- **Precision:** FP16

## Installation

Please refer to the installation instructions in our [repository](https://github.com/rajpurkarlab/ColonCrafter). 

## Usage

```python
import torch
from src.depth.models.model import ColonCrafterInference

# Load the model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = ColonCrafterInference.from_pretrained(
    "romainhardy/coloncrafter",
    device=device
)

# Prepare video tensor: (N, C, H, W) in [0, 1] range
# video = ...

# Run inference
pred_depth, pred_disparity = model.predict_depth(
    video,
    num_inference_steps=1,
    window_size=16,
    overlap=8,
    guidance_scale=1.0,
    seed=42
)
```

## Citation

If you use this model in your research, please cite:

```bibtex
@article{hardy2025coloncrafter,
  title={ColonCrafter: A Depth Estimation Model for Colonoscopy Videos Using Diffusion Priors},
  author={Hardy, Romain and Berzin, Tyler and Rajpurkar, Pranav},
  journal={arXiv preprint arXiv:2509.13525},
  year={2025}
}
```

## Acknowledgments

This model builds upon [DepthCrafter](https://github.com/Tencent/DepthCrafter) and [Stable Video Diffusion](https://huggingface.co/stabilityai/stable-video-diffusion-img2vid-xt).