README.md

metadata

title: Image → Depth → 3D Gaussian Splatting Pipeline
emoji: 🌐
colorFrom: indigo
colorTo: purple
sdk: gradio
sdk_version: 4.44.0
python_version: '3.12'
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - background-removal
  - depth-estimation
  - gaussian-splatting
  - 3d-reconstruction
  - image-segmentation
  - transformers
secrets:
  - HF_TOKEN

🌐 Image → Depth → 3DGS Pipeline

A modular, end-to-end pipeline that chains three stages on a user-supplied image:

1. Background Removal — Matte the subject out with BiRefNet (MIT), crop to it, and composite it onto a clean background before anything else runs
2. Depth Estimation — Predict a dense depth map from the background-removed image (Depth Anything v2 / MiDaS / DPT)
3. 3D Gaussian Splatting init — Convert the RGBD data into a point cloud / Gaussian splat scaffold, using the background-removal matte to cleanly separate subject from background points

Each stage uses a swappable model selected from a dropdown or custom HuggingFace model ID.

Why background removal first?

Monocular depth models are noticeably better behaved on an isolated subject than on a photo with a busy background — there's no sky/horizon to confuse the depth ordering, and the reconstruction stage can use the real segmentation matte (instead of a "is this pixel dark" brightness guess) to decide which back-projected points belong to the subject.

The background-removal stage borrows both its model family and its pre-processing recipe (predict alpha → erode the matte edge → crop to the subject's bounding box with a margin → composite onto a solid colour) from VAST-AI/TripoSplat, which uses BiRefNet for the same purpose ahead of its own Gaussian-splat generator. Rather than re-host TripoSplat's from-scratch Swin-L port and its bundled checkpoint, this project loads BiRefNet the standard way, via transformers.AutoModelForImageSegmentation, so it drops into the same "swap any HF model id" pattern as the depth and reconstruction stages.

Models

Stage	Default	Alternatives
Background Removal	ZhengPeng7/BiRefNet	BiRefNet Lite, BiRefNet Portrait, custom
Depth Estimation	depth-anything/Depth-Anything-V2-Small-hf	MiDaS v3, DPT-Large, ZoeDepth
3D Reconstruction	Point cloud from RGBD	Gaussian splat scaffold

All background-removal presets are MIT-licensed and trained on the open DIS5K dataset.

Usage

1. Upload an image
2. Select models for each stage (or paste a HuggingFace model ID)
3. Run the full pipeline or individual stages
4. Download outputs: background-removed image, depth map, point cloud PLY

Local Setup

pip install -r requirements.txt
python app.py