--- license: mit library_name: gaussianformer tags: - 3d-gaussian-splatting - neural-rendering - novel-view-synthesis - transformer - pytorch_model_hub_mixin - model_hub_mixin - gaussianformer pipeline_tag: image-to-image datasets: - ShapeSplats/Objaverse_Splats base_model: - microsoft/renderformer-v1-base --- # GaussianFormer V21 GaussianFormer renders 3D Gaussian Splatting scenes with a transformer, without per-scene optimization and with no rasterizer in the renderer. It adapts [RenderFormer](https://huggingface.co/microsoft/renderformer-v1-base) (SIGGRAPH 2025), a transformer renderer for triangle meshes: each Gaussian's 14 parameters `[pos(3), scale(3), quat(4), rgb(3), opacity(1)]` become one scene token, and the two-stage architecture (view-independent scene encoder, view-dependent ray decoder with a DPT head) is kept and warm-started from RenderFormer's weights. **Paper:** [arXiv:2611.05593](https://arxiv.org/abs/2611.05593) · **Project page:** [svlwoof.github.io/gaussianrenderformer](https://svlwoof.github.io/gaussianrenderformer/) · **Code:** [github.com/SVLwoof/gaussianformer](https://github.com/SVLwoof/gaussianformer) ## How to use Install the [code](https://github.com/SVLwoof/gaussianformer) with [uv](https://docs.astral.sh/uv/) (Linux, NVIDIA GPU, Python 3.12); this also brings PyTorch for CUDA 12.8. Flash Attention is optional and recommended for speed: the `flash` extra adds a prebuilt wheel, and without it the model uses PyTorch's scaled-dot-product attention. ```bash uv add "gaussianformer @ git+https://github.com/SVLwoof/gaussianformer" # or: uv pip install "..." uv add "gaussianformer[flash] @ git+https://github.com/SVLwoof/gaussianformer" # with Flash Attention ``` Then, from a 3D Gaussian Splatting PLY: ```python import torch from gaussianformer import GaussianFormerRenderingPipeline, load_ply from gaussianformer.utils.cameras import orbit pipeline = GaussianFormerRenderingPipeline.from_pretrained("shahafvl/gaussianformer-v21").to("cuda") gaussians = load_ply("object.ply", up="z") # [N, 14]; up = the file's up axis c2w = torch.from_numpy(orbit(8, 1.7)) # [V, 4, 4] camera-to-world (-Z forward, +Y up) images = pipeline(gaussians, c2w, fov=45.0) # [V, 512, 512, 3] ``` The model takes each Gaussian as 14 activated values, pos(3) | scale(3) | quat wxyz(4) | rgb(3) | opacity(1): linear scales, a unit quaternion, colors and opacity in [0, 1], for an object centered, Y-up and scaled into [-0.45, 0.45]^3, about 20k Gaussians. A PLY stores log scales, raw quaternions, spherical-harmonic coefficients and opacity logits; `load_ply` converts them, normalizes the object, prunes it to 20k Gaussians and fine-tunes the kept ones against the full splat (about a minute on a GPU). From the command line, in a clone of the repository (`uv sync`): ```bash uv run python -m tools.ply_to_h5 --ply object.ply --out object.h5 --up z uv run python infer.py --model shahafvl/gaussianformer-v21 --h5 object.h5 --out renders/ # one PNG per camera ``` This is the final recipe without the projected footprint: dense cross-attention, the projected 2-D RoPE and 4 px patches, trained exactly like [shahafvl/gaussianformer](https://huggingface.co/shahafvl/gaussianformer) (V22), which adds the footprint and is the recommended model. It is published for comparison. ## Results 300 objects held out from training and never used while developing the model, PSNR on the square crop around the object against the full-splat ground truth, over all 28 training views and over 4 of them (views 0, 4, 7 and 11 at camera distance 1.7): | | 28 views | 4 views | SSIM | LPIPS | |---|---|---|---|---| | gsplat rasterization of the same 20k-Gaussian input | 44.02 | 44.97 | 0.9954 | 0.0046 | | GaussianFormer V21 (this model) | 36.67 | 36.87 | 0.9779 | 0.0140 | | GaussianFormer V22 (with the footprint) | 38.08 | 38.36 | 0.9814 | 0.0113 | On 300 training objects, at their 28 training views, V21 scores 36.86 dB. ## Training - **Data:** all 26,820 training objects of [Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), each pruned from ~50k to 20k Gaussians (LightGaussian importance score, then a short recovery fine-tune of the kept Gaussians). Targets are gsplat renders of the full, unpruned splat at 512 px: 28 views per object at camera distances 1.15, 1.7 and 2.45. - **Schedule:** initialized from RenderFormer's weights and trained at 256 px (the published checkpoint [gaussianformer-256px](https://huggingface.co/shahafvl/gaussianformer-256px)), then the projected 2-D RoPE and 4 px patches switched on in a short 256 px stage, then 385k steps at 512 px on 8 GPUs (one view per GPU per step; 4 passes over every view): a warmup-stable-decay learning rate at 5e-5, with a cosine decay to 5e-7 over the last 10k steps. - **Loss:** log-HDR L1 (background pixels weighted 0.05) and LPIPS-VGG, 0.5 each. - `scripts/train.sh` in the code repository runs all stages. ## Limitations - Trained on isolated single objects centered in [-0.45, 0.45]^3, with a 45° field of view, 512x512 pixels and camera distances 1.15 to 2.45; other resolutions, fields of view and distances cost quality. Real captures work only as regions cut out of a scene and viewed from outside. - Inputs of about 20k Gaussians: denser inputs lower its quality, while they raise a rasterizer's, and the scene encoder's attention is quadratic in the number of Gaussians. - Colors are view-independent (only the DC color of each Gaussian is used). - Fine detail (thin strokes, engravings, small text) is softer than rasterizing the same splat, and the model is much slower than a rasterizer (about 0.3 s for one 512 px view of 20k Gaussians on an L40S). ## License MIT, for the weights as for the code. The model is trained on [Objaverse_Splats](https://huggingface.co/datasets/ShapeSplats/Objaverse_Splats), splats fit to [Objaverse](https://objaverse.allenai.org/) objects, each of which keeps its creator's license. ## Citation The paper ([arXiv:2611.05593](https://arxiv.org/abs/2611.05593)) calls the model GaussianRenderFormer: ```bibtex @article{valencialevy2026gaussianrenderformer, title = {GaussianRenderFormer: Transformer-based Rendering of 3D Gaussian Splats}, author = {Valencia Levy, Shahaf and Benaim, Sagie}, journal = {arXiv preprint arXiv:2611.05593}, year = {2026} } ``` GaussianFormer builds on RenderFormer; please also cite: ```bibtex @inproceedings{zeng2025renderformer, title = {RenderFormer: Transformer-based Neural Rendering of Triangle Meshes with Global Illumination}, author = {Chong Zeng and Yue Dong and Pieter Peers and Hongzhi Wu and Xin Tong}, booktitle = {ACM SIGGRAPH 2025 Conference Papers}, year = {2025} } ```