---
title: Image → Depth → 3D Gaussian Splatting Pipeline
emoji: 🌐
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 4.44.0
python_version: "3.12"
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - background-removal
  - depth-estimation
  - gaussian-splatting
  - 3d-reconstruction
  - image-segmentation
  - transformers
secrets:
  - HF_TOKEN
---

# 🌐 Image → Depth → 3DGS Pipeline

A modular, end-to-end pipeline that chains three stages on a **user-supplied image**:

1. **Background Removal** — Matte the subject out with [BiRefNet](https://huggingface.co/ZhengPeng7/BiRefNet) (MIT), crop to it, and composite it onto a clean background before anything else runs
2. **Depth Estimation** — Predict a dense depth map from the background-removed image (Depth Anything v2 / MiDaS / DPT)
3. **3D Gaussian Splatting init** — Convert the RGBD data into a point cloud / Gaussian splat scaffold, using the background-removal matte to cleanly separate subject from background points

Each stage uses a **swappable model** selected from a dropdown or custom HuggingFace model ID.

## Why background removal first?

Monocular depth models are noticeably better behaved on an isolated subject than on a
photo with a busy background — there's no sky/horizon to confuse the depth ordering, and
the reconstruction stage can use the real segmentation matte (instead of a "is this pixel
dark" brightness guess) to decide which back-projected points belong to the subject.

The background-removal stage borrows both its model family and its pre-processing recipe
(predict alpha → erode the matte edge → crop to the subject's bounding box with a margin →
composite onto a solid colour) from [VAST-AI/TripoSplat](https://huggingface.co/spaces/VAST-AI/TripoSplat),
which uses BiRefNet for the same purpose ahead of its own Gaussian-splat generator. Rather
than re-host TripoSplat's from-scratch Swin-L port and its bundled checkpoint, this project
loads BiRefNet the standard way, via `transformers.AutoModelForImageSegmentation`, so it
drops into the same "swap any HF model id" pattern as the depth and reconstruction stages.

## Models

| Stage | Default | Alternatives |
|-------|---------|-------------|
| Background Removal | `ZhengPeng7/BiRefNet` | BiRefNet Lite, BiRefNet Portrait, custom |
| Depth Estimation | `depth-anything/Depth-Anything-V2-Small-hf` | MiDaS v3, DPT-Large, ZoeDepth |
| 3D Reconstruction | Point cloud from RGBD | Gaussian splat scaffold |

All background-removal presets are MIT-licensed and trained on the open DIS5K dataset.

## Usage

1. Upload an image
2. Select models for each stage (or paste a HuggingFace model ID)
3. Run the full pipeline or individual stages
4. Download outputs: background-removed image, depth map, point cloud PLY

## Local Setup

```bash
pip install -r requirements.txt
python app.py
```