TuanNgo commited on
Commit
9463344
·
verified ·
1 Parent(s): e8db0bb

Add model card

Browse files
Files changed (1) hide show
  1. README.md +94 -0
README.md CHANGED
@@ -1,3 +1,97 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
+ library_name: volfill
4
+ pipeline_tag: image-to-3d
5
+ tags:
6
+ - 3d-reconstruction
7
+ - amodal-completion
8
+ - single-view-reconstruction
9
+ - scene-reconstruction
10
+ - flow-matching
11
+ - diffusion-transformer
12
+ - point-cloud
13
  ---
14
+
15
+ # VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching
16
+
17
+ <p align="center">
18
+ <a href="https://arxiv.org/abs/2605.31466"><img src="https://img.shields.io/badge/arXiv-2605.31466-b31b1b.svg" alt="arXiv"></a>
19
+ <a href="https://ngoductuanlhp.github.io/VolFill/"><img src="https://img.shields.io/badge/Project-Page-1f72b8.svg" alt="Project Page"></a>
20
+ <a href="https://github.com/ngoductuanlhp/VolFill"><img src="https://img.shields.io/badge/Code-GitHub-181717.svg?logo=github" alt="Code"></a>
21
+ </p>
22
+
23
+ <p align="center">
24
+ <img src="https://raw.githubusercontent.com/ngoductuanlhp/VolFill/main/assets/teaser.png" width="100%" alt="VolFill teaser">
25
+ </p>
26
+
27
+ Pretrained checkpoints for **VolFill**, which recovers the **complete** 3D scene
28
+ geometry — including occluded surfaces — from a **single RGB image**, represented
29
+ as a 256³ Truncated Unsigned Distance Function (TUDF) grid.
30
+
31
+ > Authors: Tuan Duc Ngo¹, Chuang Gan¹, Evangelos Kalogerakis¹˒²
32
+ > &nbsp;|&nbsp; ¹University of Massachusetts Amherst &nbsp; ²Technical University of Crete
33
+
34
+ ## Model description
35
+
36
+ VolFill is a two-stage latent generative model. A **hybrid 3D VAE** (sparse
37
+ encoder → dense bottleneck → hybrid dense-to-sparse decoder) compresses the 256³
38
+ TUDF to a compact 16³×16ch latent, and a **latent Diffusion Transformer trained
39
+ with flow matching** generates that latent — conditioned on (a) frozen MoGe-v2
40
+ image features as a global geometric prior and (b) a visible-geometry latent that
41
+ anchors the occluded regions. At inference the model encodes the visible region,
42
+ samples the DiT for 50 Euler steps with CFG = 3.0, and decodes to a TUDF that is
43
+ thresholded into a point cloud or mesh.
44
+
45
+ ## Files
46
+
47
+ | File | Description |
48
+ |---|---|
49
+ | `volfill_dit.pth` | Latent flow-matching DiT (visible-latent conditioned, 16× variant) |
50
+ | `volfill_vae.pth` | Hybrid 3D VAE (sparse encoder + hybrid decoder) |
51
+ | `inference.yaml` | Model architecture + sampler config |
52
+ | `latent_stats_16x.npy` | Latent normalization statistics (mean / std) |
53
+
54
+ The MoGe geometry prior (`Ruicheng/moge-2-vitl`, `Ruicheng/moge-2-vitl-normal`)
55
+ is downloaded automatically on first run.
56
+
57
+ ## Usage
58
+
59
+ Install the inference code from the [GitHub repo](https://github.com/ngoductuanlhp/VolFill)
60
+ (CUDA 13.0 / RTX 40-series), then everything in this model repo downloads
61
+ automatically:
62
+
63
+ ```bash
64
+ # CLI — all weights/config/stats auto-download from this repo
65
+ python -m volfill.amodal.inference_latent_visible \
66
+ --hf_repo TuanNgo/VolFill --input_path image.jpg --output ./results/
67
+ ```
68
+
69
+ ```python
70
+ from PIL import Image
71
+ from volfill.amodal.inference_latent_visible import LatentTUDFVisibleInference
72
+
73
+ infer = LatentTUDFVisibleInference.from_pretrained("TuanNgo/VolFill")
74
+ result = infer(Image.open("image.jpg").convert("RGB"))
75
+ # result["tudf"]: (1, 1, 256, 256, 256) predicted TUDF in [-1, 1]
76
+ ```
77
+
78
+ See the GitHub README for installation, point-cloud visualization, and local /
79
+ Google-Drive checkpoint options.
80
+
81
+ ## Citation
82
+
83
+ ```bibtex
84
+ @article{ngo2026volfill,
85
+ title = {VolFill: Single-View Amodal 3D Scene Reconstruction with Volumetric Flow Matching},
86
+ author = {Ngo, Tuan Duc and Gan, Chuang and Kalogerakis, Evangelos},
87
+ journal = {arXiv preprint arXiv:2605.31466},
88
+ year = {2026}
89
+ }
90
+ ```
91
+
92
+ ## License & acknowledgements
93
+
94
+ Released under the MIT License. Built on
95
+ [LaRI](https://github.com/ruili3/LaRI), reuses sparse-conv modules from
96
+ [TRELLIS](https://github.com/microsoft/TRELLIS), and uses
97
+ [MoGe-v2](https://github.com/microsoft/MoGe) as the visible geometry prior.