Add model card: pipeline tag, library name, and paper link
Browse filesCreates a model card for the DiffusionSfM checkpoint (ckpt_00800000.pth), which previously had no README.
Changes:
- Add pipeline_tag: image-to-3d (multi-view images -> 3D scene geometry + camera poses)
- Add library_name: pytorch (raw .pth checkpoint, custom PyTorch stack; no transformers/diffusers artifacts)
- Add paper link to https://huggingface.co/papers/2505.05473 (CVPR 2025)
- Add license: mit (from the official GitHub repo LICENSE)
- Add brief model description, intended uses, and a minimal, verified usage snippet using huggingface_hub.hf_hub_download
- Link to the project page, GitHub repo, and Gradio demo Space
README.md
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: mit
|
| 3 |
+
library_name: pytorch
|
| 4 |
+
pipeline_tag: image-to-3d
|
| 5 |
+
tags:
|
| 6 |
+
- structure-from-motion
|
| 7 |
+
- multi-view
|
| 8 |
+
- pose-estimation
|
| 9 |
+
- 3d-reconstruction
|
| 10 |
+
- diffusion
|
| 11 |
+
- computer-vision
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
|
| 15 |
+
# DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion
|
| 16 |
+
|
| 17 |
+
This repository hosts the pretrained checkpoint for **DiffusionSfM**, a transformer-based denoising diffusion model that directly infers 3D scene geometry and camera poses from multi-view images. The model parameterizes scene geometry and cameras as pixel-wise ray origins and endpoints in a global frame and learns a denoising diffusion model to predict them from multi-view inputs.
|
| 18 |
+
|
| 19 |
+
## Paper
|
| 20 |
+
|
| 21 |
+
**DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion**
|
| 22 |
+
Qitao Zhao, Amy Lin, Jeff Tan, Jason Y. Zhang, Deva Ramanan, Shubham Tulsiani
|
| 23 |
+
Accepted to CVPR 2025.
|
| 24 |
+
|
| 25 |
+
For more details, see the paper: https://huggingface.co/papers/2505.05473
|
| 26 |
+
|
| 27 |
+
- arXiv: https://arxiv.org/abs/2505.05473
|
| 28 |
+
- Project page: https://qitaozhao.github.io/DiffusionSfM
|
| 29 |
+
- Code: https://github.com/QitaoZhao/DiffusionSfM
|
| 30 |
+
- Interactive demo (Gradio Space): https://huggingface.co/spaces/qitaoz/DiffusionSfM
|
| 31 |
+
|
| 32 |
+
## Model description
|
| 33 |
+
|
| 34 |
+
DiffusionSfM is a data-driven Structure-from-Motion (SfM) approach that replaces the conventional two-stage pipeline (pairwise reasoning + global optimization) with a single end-to-end multi-view reasoning model. Given a set of multi-view images, it predicts, per pixel, the ray origin and ray endpoint in a global coordinate frame. The endpoints give the 3D scene geometry (point structure), while the ray origins and endpoints together recover the camera poses. A transformer-based denoising diffusion model performs this prediction directly from multi-view image features. Specialized training mechanisms handle missing data and unbounded scene coordinates. The model naturally models uncertainty through the diffusion process and outperforms classical and prior learning-based SfM approaches on both synthetic and real datasets.
|
| 35 |
+
|
| 36 |
+
This repository contains the released checkpoint `ckpt_00800000.pth` (a raw PyTorch checkpoint). It is intended to be used together with the official codebase at https://github.com/QitaoZhao/DiffusionSfM.
|
| 37 |
+
|
| 38 |
+
## Intended uses
|
| 39 |
+
|
| 40 |
+
- Multi-view Structure-from-Motion (camera pose estimation + 3D structure recovery) from a set of posed or unposed images.
|
| 41 |
+
- Research on learned SfM, multi-view geometry, and diffusion models for 3D vision.
|
| 42 |
+
|
| 43 |
+
## How to use
|
| 44 |
+
|
| 45 |
+
The checkpoint is a raw PyTorch `.pth` file and is loaded by the official codebase. It is not directly loadable with `transformers` or `diffusers` `from_pretrained` APIs; use the project's own loading utilities.
|
| 46 |
+
|
| 47 |
+
1. Clone the official codebase:
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
git clone https://github.com/QitaoZhao/DiffusionSfM.git
|
| 51 |
+
cd DiffusionSfM
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
2. Download the checkpoint from this repo:
|
| 55 |
+
|
| 56 |
+
```python
|
| 57 |
+
from huggingface_hub import hf_hub_download
|
| 58 |
+
|
| 59 |
+
ckpt_path = hf_hub_download(repo_id="qitaoz/DiffusionSfM", filename="ckpt_00800000.pth")
|
| 60 |
+
```
|
| 61 |
+
|
| 62 |
+
3. Follow the installation and inference instructions in the [official repository](https://github.com/QitaoZhao/DiffusionSfM) (environment setup with `torch`, `torchvision`, `pytorch3d`, and the project's `diffusionsfm` package). You can also try the online [Gradio demo](https://huggingface.co/spaces/qitaoz/DiffusionSfM) without any local setup.
|
| 63 |
+
|
| 64 |
+
## License
|
| 65 |
+
|
| 66 |
+
This model is released under the MIT License (see the [LICENSE](https://github.com/QitaoZhao/DiffusionSfM/blob/main/LICENSE) in the official repository).
|
| 67 |
+
|
| 68 |
+
## Citation
|
| 69 |
+
|
| 70 |
+
If you use this model, please cite:
|
| 71 |
+
|
| 72 |
+
```bibtex
|
| 73 |
+
@inproceedings{zhao2025diffusionsfm,
|
| 74 |
+
title={DiffusionSfM: Predicting Structure and Motion via Ray Origin and Endpoint Diffusion},
|
| 75 |
+
author={Qitao Zhao and Amy Lin and Jeff Tan and Jason Y. Zhang and Deva Ramanan and Shubham Tulsiani},
|
| 76 |
+
booktitle={CVPR},
|
| 77 |
+
year={2025}
|
| 78 |
+
}
|
| 79 |
+
```
|