Blame for README.md

---
license: mit
library_name: gaussianformer
tags:
- 3d-gaussian-splatting
- neural-rendering
- novel-view-synthesis
- transformer
- pytorch_model_hub_mixin
- model_hub_mixin
- gaussianformer
pipeline_tag: image-to-image
datasets:
- ShapeSplats/Objaverse_Splats
base_model:
- microsoft/renderformer-v1-base
---

# GaussianFormer V21

GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization and with no
rasterizer in the renderer. It adapts [RenderFormer](https://huggingface.co/microsoft/renderformer-v1-base) (SIGGRAPH
2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters `[pos(3), scale(3), quat(4), rgb(3),
opacity(1)]` become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent
ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights.

**Paper:** [arXiv:2611.05593](https://arxiv.org/abs/2611.05593) · **Project page:** [svlwoof.github.io/gaussianrenderformer](https://svlwoof.github.io/gaussianrenderformer/) · **Code:** [github.com/SVLwoof/gaussianformer](https://github.com/SVLwoof/gaussianformer)

## How to use

Install the [code](https://github.com/SVLwoof/gaussianformer) with [uv](https://docs.astral.sh/uv/) (Linux, NVIDIA GPU,
Python 3.12); this also brings PyTorch for CUDA 12.8. Flash Attention is optional and recommended for speed: the
`flash` extra adds a prebuilt wheel, and without it the model uses PyTorch's scaled-dot-product attention.

```bash
uv add "gaussianformer @ git+https://github.com/SVLwoof/gaussianformer"            # or: uv pip install "..."
uv add "gaussianformer[flash] @ git+https://github.com/SVLwoof/gaussianformer"     # with Flash Attention
```

Then, from a 3D Gaussian Splatting PLY:

```python
import torch
from gaussianformer import GaussianFormerRenderingPipeline, load_ply
from gaussianformer.utils.cameras import orbit

pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v21").to("cuda")
gaussians = load_ply("object.ply", up="z")     # [N, 14]; up = the file's up axis
c2w = torch.from_numpy(orbit(8, 1.7))          # [V, 4, 4] camera-to-world (-Z forward, +Y up)
images = pipeline(gaussians, c2w, fov=45.0)    # [V, 512, 512, 3]
```

The model takes each Gaussian as 14 activated values, pos(3) | scale(3) | quat wxyz(4) | rgb(3) | opacity(1): linear
scales, a unit quaternion, colors and opacity in [0, 1], for an object centered, Y-up and scaled into [-0.45, 0.45]^3,
about 20k Gaussians. A PLY stores log scales, raw quaternions, spherical-harmonic coefficients and opacity logits;
`load_ply` converts them, normalizes the object, prunes it to 20k Gaussians and fine-tunes the kept ones
against the full splat (about a minute on a GPU).

From the command line, in a clone of the repository (`uv sync`):

```bash
uv run python -m tools.ply_to_h5 --ply object.ply --out object.h5 --up z
uv run python infer.py --model shahafvl/gaussianformer-v21 --h5 object.h5 --out renders/   # one PNG per camera
```

This is the final recipe without the projected footprint: dense cross-attention, the projected 2-D RoPE and 4 px
patches, trained exactly like [shahafvl/gaussianformer](https://huggingface.co/shahafvl/gaussianformer) (V22), which
adds the footprint and is the recommended model. It is published for comparison.

## Results

300 objects held out from training and never used while developing the model, PSNR on the square crop around the
object against the full-splat ground truth, over all 28 training views and over 4 of them (views 0, 4, 7 and 11 at
camera distance 1.7):

| | 28 views | 4 views | SSIM | LPIPS |
|---|---|---|---|---|
| gsplat rasterization of the same 20k-Gaussian input | 44.02 | 44.97 | 0.9954 | 0.0046 |
| GaussianFormer V21 (this model) | 36.67 | 36.87 | 0.9779 | 0.0140 |
| GaussianFormer V22 (with the footprint) | 38.08 | 38.36 | 0.9814 | 0.0113 |

On 300 training objects, at their 28 training views, V21 scores 36.86 dB.

## Training

- **Data:** all 26,820 training objects of
  [Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), each pruned
  from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians).
  Targets are gsplat renders of the full, unpruned splat at 512 px: 28 views per object at camera distances 1.15,
  1.7 and 2.45.
- **Schedule:** initialized from RenderFormer's weights and trained at 256 px (the published checkpoint
  [gaussianformer-256px](https://huggingface.co/shahafvl/gaussianformer-256px)), then the projected 2-D RoPE and 4 px patches switched on in a short
  256 px stage, then 385k steps at 512 px on 8 GPUs (one view per GPU per step; 4 passes over every view): a
  warmup-stable-decay learning rate at 5e-5, with a cosine decay to 5e-7 over the last 10k steps.
- **Loss:** log-HDR L1 (background pixels weighted 0.05) and LPIPS-VGG, 0.5 each.
- `scripts/train.sh` in the code repository runs all stages.

## Limitations

- Trained on isolated single objects centered in [-0.45, 0.45]^3, with a 45° field of view, 512x512 pixels and
  camera distances 1.15 to 2.45; other resolutions, fields of view and distances cost quality. Real captures work
  only as regions cut out of a scene and viewed from outside.
- Inputs of about 20k Gaussians: denser inputs lower its quality, while they raise a rasterizer's, and the scene
  encoder's attention is quadratic in the number of Gaussians.
- Colors are view-independent (only the DC color of each Gaussian is used).
- Fine detail (thin strokes, engravings, small text) is softer than rasterizing the same splat, and the model is
  much slower than a rasterizer (about 0.3 s for one 512 px view of 20k Gaussians on an L40S).

## License

MIT, for the weights as for the code. The model is trained on
[Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), splats fit to
[Objaverse](https://objaverse.allenai.org/) objects, each of which keeps its creator's license.

## Citation

The paper ([arXiv:2611.05593](https://arxiv.org/abs/2611.05593)) calls the model GaussianRenderFormer:

```bibtex
@article{valencialevy2026gaussianrenderformer,
  title     = {GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats},
  author    = {Valencia Levy, Shahaf and Benaim, Sagie},
  journal   = {arXiv preprint arXiv:2611.05593},
  year      = {2026}
}
```

GaussianFormer builds on RenderFormer; please also cite:

```bibtex
@inproceedings{zeng2025renderformer,
  title     = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination},
  author    = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong},
  booktitle = {ACM SIGGRAPH 2025 Conference Papers},
  year      = {2025}
}
```