license: mit
base_model: microsoft/renderformer-v1-base
tags:
- rendering
- global-illumination
- sparse-attention
- neural-rendering
library_name: pytorch
SparseRenderFormer (GP-Sparse)
A retrofit of RenderFormer that replaces the dense view-independent self-attention with a block-sparse pattern derived from the scene's bounding volume hierarchy.
The idea
Sparse attention normally gets its block structure from sequence position: nearby tokens are likely to interact. RenderFormer's view-independent stage is permutation invariant, so index adjacency says nothing about whether two triangles exchange light. The block structure is therefore built from geometry instead: a median-split BVH over triangle centroids permutes triangles so each leaf is a contiguous run of slots, which lets leaf index serve as kernel block index.
Three branches are unioned into one mask and evaluated by a single fused kernel:
- coarse — each leaf pooled by triangle area into one token, visible to every query. Not optional: without it a triangle cannot see a light source outside its own neighbourhood, and training cannot recover a token it never attends to.
- selected — a learned scorer retrieves additional leaves per query block.
- local — the nearest leaves by block-centroid distance.
The differential form factor A_j cos(t_i) cos(t_j) / r^2 is added to the logits
of retained pairs as a score modifier: no parameters, under 0.5% of runtime.
Speed
View-independent stage, A100 80GB, bfloat16, batch 1, 12 layers:
| N | dense | this model | speed-up |
|---|---|---|---|
| 8,192 | 14.2 ms / 1.47 GB | 25.5 ms / 0.37 GB | 0.6x |
| 16,384 | 56.5 ms / 1.77 GB | 44.4 ms / 0.58 GB | 1.3x |
| 32,768 | 220.9 ms / 2.38 GB | 87.4 ms / 0.98 GB | 2.5x |
Crossover is near N = 14,000. Below it this model is slower than dense.
Note these cover the attention stage only. The feed-forward network is O(N) and does not shrink, so the model-level speed-up is lower and is not yet measured.
Quality, and its limits
Fine-tuned 3,000 steps against Blender Cycles references. 19.78 -> 27.35 dB, where dense RenderFormer scores 31.23 dB on the same references: a deficit of about 3.3 dB.
Two caveats that bound this number, both important:
- Quality was measured where the method is slower. Every scene is at most 3,841 triangles, below the N = 14,000 crossover. Quality at 16k and 32k, where the speed-up applies, is not yet measured.
- The deficit is dominated by training-set size, not by sparsity. A controlled sweep varying only the number of distinct training scenes moved PSNR by 4.53 dB: 58 scenes plateaus within 250 steps, while 250 and 1,000 scenes were still improving when the budget ended. This checkpoint was trained on 58 scenes.
Warm start
GP-Sparse keeps RenderFormer's query, key, value and output projections and
changes only which keys each query may see. Splitting the pretrained fused
in_proj into separate Q, K and V transfers 99.27% of the checkpoint
(205.2M of 206.7M parameters); only the block scorer and branch gates are new.
This bounds the claim: it shows sparse attention can recover dense behaviour cheaply, not that it is natively trainable to that quality from scratch.
Files
student_bf16.pt— bfloat16 weights, optimiser state stripped. Inference here runs under bfloat16 autocast, so this loses nothing that is used.
Loading
import torch
from huggingface_hub import hf_hub_download
from renderformer.models.config import RenderFormerConfig
from renderformer.models.renderformer import RenderFormer
path = hf_hub_download("douyeszn/sparserenderformer-gp-sparse", "student_bf16.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
model = RenderFormer(RenderFormerConfig(**ckpt["config"]))
model.load_state_dict(ckpt["model_state_dict"])
Code: the scene generator, Cycles pipeline and every experiment script are released alongside this model.
Data
Trained against douyeszn/sparserenderformer-cycles,
path-traced references in RenderFormer's exact HDF5 schema. RenderFormer's own
2M-scene training set was never published.
Citing
This is work in progress; a preprint is not yet posted. Please cite RenderFormer (Zeng et al., SIGGRAPH 2025) for the base model.