README.md

---
license: mit
base_model: microsoft/renderformer-v1-base
tags:
- rendering
- global-illumination
- sparse-attention
- neural-rendering
library_name: pytorch
---

# SparseRenderFormer (GP-Sparse)

A retrofit of [RenderFormer](https://huggingface.co/microsoft/renderformer-v1-base)
that replaces the dense view-independent self-attention with a **block-sparse
pattern derived from the scene's bounding volume hierarchy**.

## The idea

Sparse attention normally gets its block structure from sequence position:
nearby tokens are likely to interact. RenderFormer's view-independent stage is
**permutation invariant**, so index adjacency says nothing about whether two
triangles exchange light. The block structure is therefore built from geometry
instead: a median-split BVH over triangle centroids permutes triangles so each
leaf is a contiguous run of slots, which lets leaf index serve as kernel block
index.

Three branches are unioned into one mask and evaluated by a single fused kernel:

- **coarse** — each leaf pooled by triangle area into one token, visible to every
  query. Not optional: without it a triangle cannot see a light source outside
  its own neighbourhood, and training cannot recover a token it never attends to.
- **selected** — a learned scorer retrieves additional leaves per query block.
- **local** — the nearest leaves by block-centroid distance.

The differential form factor `A_j cos(t_i) cos(t_j) / r^2` is added to the logits
of retained pairs as a score modifier: no parameters, under 0.5% of runtime.

## Speed

View-independent stage, A100 80GB, bfloat16, batch 1, 12 layers:

| N | dense | this model | speed-up |
|---|---|---|---|
| 8,192 | 14.2 ms / 1.47 GB | 25.5 ms / 0.37 GB | 0.6x |
| 16,384 | 56.5 ms / 1.77 GB | 44.4 ms / 0.58 GB | 1.3x |
| 32,768 | 220.9 ms / 2.38 GB | 87.4 ms / 0.98 GB | **2.5x** |

Crossover is near N = 14,000. **Below it this model is slower than dense.**

Note these cover the attention stage only. The feed-forward network is O(N) and
does not shrink, so the model-level speed-up is lower and is not yet measured.

## Quality, and its limits

Fine-tuned 3,000 steps against Blender Cycles references. **19.78 -> 27.35 dB**,
where dense RenderFormer scores **31.23 dB** on the same references: a deficit of
about 3.3 dB.

Two caveats that bound this number, both important:

1. **Quality was measured where the method is slower.** Every scene is at most
   3,841 triangles, below the N = 14,000 crossover. Quality at 16k and 32k, where
   the speed-up applies, is not yet measured.
2. **The deficit is dominated by training-set size, not by sparsity.** A
   controlled sweep varying only the number of distinct training scenes moved
   PSNR by **4.53 dB**: 58 scenes plateaus within 250 steps, while 250 and 1,000
   scenes were still improving when the budget ended. This checkpoint was trained
   on 58 scenes.

## Warm start

GP-Sparse keeps RenderFormer's query, key, value and output projections and
changes only *which keys each query may see*. Splitting the pretrained fused
`in_proj` into separate Q, K and V transfers **99.27%** of the checkpoint
(205.2M of 206.7M parameters); only the block scorer and branch gates are new.

This bounds the claim: it shows sparse attention can *recover* dense behaviour
cheaply, not that it is natively trainable to that quality from scratch.

## Files

- `student_bf16.pt` — bfloat16 weights, optimiser state stripped. Inference here
  runs under bfloat16 autocast, so this loses nothing that is used.

## Loading

```python
import torch
from huggingface_hub import hf_hub_download
from renderformer.models.config import RenderFormerConfig
from renderformer.models.renderformer import RenderFormer

path = hf_hub_download("douyeszn/sparserenderformer-gp-sparse", "student_bf16.pt")
ckpt = torch.load(path, map_location="cpu", weights_only=False)
model = RenderFormer(RenderFormerConfig(**ckpt["config"]))
model.load_state_dict(ckpt["model_state_dict"])
```

Code: the scene generator, Cycles pipeline and every experiment script are
released alongside this model.

## Data

Trained against [`douyeszn/sparserenderformer-cycles`](https://huggingface.co/datasets/douyeszn/sparserenderformer-cycles),
path-traced references in RenderFormer's exact HDF5 schema. RenderFormer's own
2M-scene training set was never published.

## Citing

This is work in progress; a preprint is not yet posted. Please cite
RenderFormer (Zeng et al., SIGGRAPH 2025) for the base model.