| --- |
| license: mit |
| library_name: gaussianformer |
| tags: |
| - 3d-gaussian-splatting |
| - neural-rendering |
| - novel-view-synthesis |
| - transformer |
| - pytorch_model_hub_mixin |
| - model_hub_mixin |
| - gaussianformer |
| pipeline_tag: image-to-image |
| datasets: |
| - ShapeSplats/Objaverse_Splats |
| base_model: |
| - microsoft/renderformer-v1-base |
| --- |
| |
| # GaussianFormer V21 |
|
|
| GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization and with no |
| rasterizer in the renderer. It adapts [RenderFormer](https://huggingface.co/microsoft/renderformer-v1-base) (SIGGRAPH |
| 2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters `[pos(3), scale(3), quat(4), rgb(3), |
| opacity(1)]` become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent |
| ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights. |
|
|
| **Paper:** [arXiv:2611.05593](https://arxiv.org/abs/2611.05593) · **Project page:** [svlwoof.github.io/gaussianrenderformer](https://svlwoof.github.io/gaussianrenderformer/) · **Code:** [github.com/SVLwoof/gaussianformer](https://github.com/SVLwoof/gaussianformer) |
|
|
| ## How to use |
|
|
| Install the [code](https://github.com/SVLwoof/gaussianformer) with [uv](https://docs.astral.sh/uv/) (Linux, NVIDIA GPU, |
| Python 3.12); this also brings PyTorch for CUDA 12.8. Flash Attention is optional and recommended for speed: the |
| `flash` extra adds a prebuilt wheel, and without it the model uses PyTorch's scaled-dot-product attention. |
|
|
| ```bash |
| uv add "gaussianformer @ git+https://github.com/SVLwoof/gaussianformer" # or: uv pip install "..." |
| uv add "gaussianformer[flash] @ git+https://github.com/SVLwoof/gaussianformer" # with Flash Attention |
| ``` |
|
|
| Then, from a 3D Gaussian Splatting PLY: |
|
|
| ```python |
| import torch |
| from gaussianformer import GaussianFormerRenderingPipeline, load_ply |
| from gaussianformer.utils.cameras import orbit |
| |
| pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v21").to("cuda") |
| gaussians = load_ply("object.ply", up="z") # [N, 14]; up = the file's up axis |
| c2w = torch.from_numpy(orbit(8, 1.7)) # [V, 4, 4] camera-to-world (-Z forward, +Y up) |
| images = pipeline(gaussians, c2w, fov=45.0) # [V, 512, 512, 3] |
| ``` |
|
|
| The model takes each Gaussian as 14 activated values, pos(3) | scale(3) | quat wxyz(4) | rgb(3) | opacity(1): linear |
| scales, a unit quaternion, colors and opacity in [0, 1], for an object centered, Y-up and scaled into [-0.45, 0.45]^3, |
| about 20k Gaussians. A PLY stores log scales, raw quaternions, spherical-harmonic coefficients and opacity logits; |
| `load_ply` converts them, normalizes the object, prunes it to 20k Gaussians and fine-tunes the kept ones |
| against the full splat (about a minute on a GPU). |
|
|
| From the command line, in a clone of the repository (`uv sync`): |
|
|
| ```bash |
| uv run python -m tools.ply_to_h5 --ply object.ply --out object.h5 --up z |
| uv run python infer.py --model shahafvl/gaussianformer-v21 --h5 object.h5 --out renders/ # one PNG per camera |
| ``` |
|
|
| This is the final recipe without the projected footprint: dense cross-attention, the projected 2-D RoPE and 4 px |
| patches, trained exactly like [shahafvl/gaussianformer](https://huggingface.co/shahafvl/gaussianformer) (V22), which |
| adds the footprint and is the recommended model. It is published for comparison. |
|
|
| ## Results |
|
|
| 300 objects held out from training and never used while developing the model, PSNR on the square crop around the |
| object against the full-splat ground truth, over all 28 training views and over 4 of them (views 0, 4, 7 and 11 at |
| camera distance 1.7): |
|
|
| | | 28 views | 4 views | SSIM | LPIPS | |
| |---|---|---|---|---| |
| | gsplat rasterization of the same 20k-Gaussian input | 44.02 | 44.97 | 0.9954 | 0.0046 | |
| | GaussianFormer V21 (this model) | 36.67 | 36.87 | 0.9779 | 0.0140 | |
| | GaussianFormer V22 (with the footprint) | 38.08 | 38.36 | 0.9814 | 0.0113 | |
|
|
| On 300 training objects, at their 28 training views, V21 scores 36.86 dB. |
|
|
| ## Training |
|
|
| - **Data:** all 26,820 training objects of |
| [Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), each pruned |
| from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). |
| Targets are gsplat renders of the full, unpruned splat at 512 px: 28 views per object at camera distances 1.15, |
| 1.7 and 2.45. |
| - **Schedule:** initialized from RenderFormer's weights and trained at 256 px (the published checkpoint |
| [gaussianformer-256px](https://huggingface.co/shahafvl/gaussianformer-256px)), then the projected 2-D RoPE and 4 px patches switched on in a short |
| 256 px stage, then 385k steps at 512 px on 8 GPUs (one view per GPU per step; 4 passes over every view): a |
| warmup-stable-decay learning rate at 5e-5, with a cosine decay to 5e-7 over the last 10k steps. |
| - **Loss:** log-HDR L1 (background pixels weighted 0.05) and LPIPS-VGG, 0.5 each. |
| - `scripts/train.sh` in the code repository runs all stages. |
|
|
| ## Limitations |
|
|
| - Trained on isolated single objects centered in [-0.45, 0.45]^3, with a 45° field of view, 512x512 pixels and |
| camera distances 1.15 to 2.45; other resolutions, fields of view and distances cost quality. Real captures work |
| only as regions cut out of a scene and viewed from outside. |
| - Inputs of about 20k Gaussians: denser inputs lower its quality, while they raise a rasterizer's, and the scene |
| encoder's attention is quadratic in the number of Gaussians. |
| - Colors are view-independent (only the DC color of each Gaussian is used). |
| - Fine detail (thin strokes, engravings, small text) is softer than rasterizing the same splat, and the model is |
| much slower than a rasterizer (about 0.3 s for one 512 px view of 20k Gaussians on an L40S). |
|
|
| ## License |
|
|
| MIT, for the weights as for the code. The model is trained on |
| [Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), splats fit to |
| [Objaverse](https://objaverse.allenai.org/) objects, each of which keeps its creator's license. |
|
|
| ## Citation |
|
|
| The paper ([arXiv:2611.05593](https://arxiv.org/abs/2611.05593)) calls the model GaussianRenderFormer: |
|
|
| ```bibtex |
| @article{valencialevy2026gaussianrenderformer, |
| title = {GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats}, |
| author = {Valencia Levy, Shahaf and Benaim, Sagie}, |
| journal = {arXiv preprint arXiv:2611.05593}, |
| year = {2026} |
| } |
| ``` |
|
|
| GaussianFormer builds on RenderFormer; please also cite: |
|
|
| ```bibtex |
| @inproceedings{zeng2025renderformer, |
| title = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination}, |
| author = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong}, |
| booktitle = {ACM SIGGRAPH 2025 Conference Papers}, |
| year = {2025} |
| } |
| ``` |
|
|
