Release weights, config and model card
Browse files- README.md +75 -0
- config.json +23 -0
- model.safetensors +3 -0
README.md
ADDED
|
@@ -0,0 +1,75 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: other
|
| 3 |
+
license_name: fair-noncommercial-research-license
|
| 4 |
+
license_link: https://github.com/Jingong-Chen/GradSplat/blob/main/LICENSE
|
| 5 |
+
library_name: gradsplat
|
| 6 |
+
pipeline_tag: image-to-3d
|
| 7 |
+
tags:
|
| 8 |
+
- gaussian-splatting
|
| 9 |
+
- novel-view-synthesis
|
| 10 |
+
- 3d-reconstruction
|
| 11 |
+
- feed-forward
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# GradSplat
|
| 15 |
+
|
| 16 |
+
**GradSplat** reconstructs a scene as 3D Gaussians from a few images in one forward pass, and refines its own prediction
|
| 17 |
+
by learned gradient steps on the input views inside that pass, first on opacity and color and then on the shape of the
|
| 18 |
+
Gaussians. No target view is used, so the steps run unchanged at inference.
|
| 19 |
+
|
| 20 |
+
[Paper](https://arxiv.org/abs/2610.38951) · [Code](https://github.com/Jingong-Chen/GradSplat) · [Project page](https://jingong-chen.github.io/GradSplat/)
|
| 21 |
+
|
| 22 |
+
These are the weights of the paper, trained on RealEstate10K. The DL3DV fine-tuned weights are at
|
| 23 |
+
[Jingong/GradSplat-DL3DV](https://huggingface.co/Jingong/GradSplat-DL3DV).
|
| 24 |
+
|
| 25 |
+
## Usage
|
| 26 |
+
|
| 27 |
+
```bash
|
| 28 |
+
git clone https://github.com/Jingong-Chen/GradSplat.git && cd GradSplat && pip install -e .
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
```python
|
| 32 |
+
from gradsplat import GradSplat
|
| 33 |
+
from gradsplat.io import load_views
|
| 34 |
+
|
| 35 |
+
model = GradSplat.from_pretrained("Jingong/GradSplat").to("cuda").eval()
|
| 36 |
+
views = load_views("examples/re10k_dining_room/images", cameras="examples/re10k_dining_room/cameras.json")
|
| 37 |
+
out = model.infer(views.images, views.c2w, views.K) # without cameras: model.infer(views.images)
|
| 38 |
+
gaussians = out["gaussians"] # one Gaussian per input pixel
|
| 39 |
+
```
|
| 40 |
+
|
| 41 |
+
## Model
|
| 42 |
+
|
| 43 |
+
- 1.22 B parameters: a VGGT-Ω backbone, a camera head, a dense head with a Gaussian branch, a shape head and two
|
| 44 |
+
learned updates (three appearance steps and five shape steps).
|
| 45 |
+
- Input: 2 or more overlapping views of 224 × 224 pixels (center-cropped), with or without their cameras.
|
| 46 |
+
- Output: one 3D Gaussian per input pixel (center, scale, rotation, opacity and spherical harmonics of degree 1), and
|
| 47 |
+
the predicted cameras.
|
| 48 |
+
- Training: initialized from VGGT-Ω and trained for 178,000 steps in four stages on the training split of
|
| 49 |
+
RealEstate10K, with 2 to 12 input views per sample.
|
| 50 |
+
|
| 51 |
+
## Results (from the paper)
|
| 52 |
+
|
| 53 |
+
| | Narrow protocol (7,128 scenes) | Wide protocol (1,600 scenes) |
|
| 54 |
+
|:---|:---:|:---:|
|
| 55 |
+
| PSNR / SSIM / LPIPS, with input poses | 33.683 / 0.9593 / 0.0640 | 26.755 / 0.8993 / 0.1301 |
|
| 56 |
+
| PSNR / SSIM / LPIPS, without input poses | 26.914 / 0.8863 / 0.1053 | 20.901 / 0.6945 / 0.2426 |
|
| 57 |
+
|
| 58 |
+
RealEstate10K with six input and eight target views per scene at 224 × 224 pixels. Without fine-tuning: DL3DV
|
| 59 |
+
24.787 / 0.8448 / 0.1919 and ScanNet++ 21.609 / 0.7540 / 0.3184.
|
| 60 |
+
|
| 61 |
+
## License
|
| 62 |
+
|
| 63 |
+
FAIR Noncommercial Research License, inherited from VGGT-Ω: noncommercial research use only.
|
| 64 |
+
|
| 65 |
+
## Citation
|
| 66 |
+
|
| 67 |
+
```bibtex
|
| 68 |
+
@article{chen2026gradsplat,
|
| 69 |
+
title = {GradSplat: Gradients Inside the Forward Pass for Feed-Forward 3D Gaussian Splatting},
|
| 70 |
+
author = {Chen, Jingong and Jiang, Linlian and Li, Guangyuan and Zuo, Xinxin and
|
| 71 |
+
Wang, Yang and Pack, Chulwoo and Gao, Kyle and Won, Kwanghee},
|
| 72 |
+
journal = {arXiv preprint arXiv:2610.38951},
|
| 73 |
+
year = {2026}
|
| 74 |
+
}
|
| 75 |
+
```
|
config.json
ADDED
|
@@ -0,0 +1,23 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"resolution": 224,
|
| 3 |
+
"patch_size": 16,
|
| 4 |
+
"embed_dim": 1024,
|
| 5 |
+
"sh_degree": 1,
|
| 6 |
+
"scale_max": 0.3,
|
| 7 |
+
"scale_mult": 0.001,
|
| 8 |
+
"dmean_scale": 0.05,
|
| 9 |
+
"gauss_upsample": 2,
|
| 10 |
+
"app_trunk_depth": 6,
|
| 11 |
+
"shape_hidden": 128,
|
| 12 |
+
"shape_amp": 4.0,
|
| 13 |
+
"shape_rot_amp": 1.0,
|
| 14 |
+
"appearance_steps": 3,
|
| 15 |
+
"shape_steps": 5,
|
| 16 |
+
"appearance_update_hidden": 192,
|
| 17 |
+
"shape_update_hidden": 128,
|
| 18 |
+
"mixer_depth": 4,
|
| 19 |
+
"mixer_dim": 768,
|
| 20 |
+
"mixer_heads": 12,
|
| 21 |
+
"mixer_mid": 64,
|
| 22 |
+
"kappa": 4.04
|
| 23 |
+
}
|
model.safetensors
ADDED
|
@@ -0,0 +1,3 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:d2bbd05a8144504c0495fed1759d5a551f4db176d18ffa378168205dc2463b76
|
| 3 |
+
size 4900169572
|