README.md

---
license: apache-2.0
base_model: Wan-AI/Wan2.1-T2V-1.3B-Diffusers
library_name: diffusers
pipeline_tag: video-to-video
tags:
  - g-buffer
  - neural-rendering
  - real-time
  - autoregressive
  - video-diffusion
  - supertuxkart
---

# AlayaRenderer-Flash

AlayaRenderer-Flash is a real-time generative world renderer. A game engine
exports G-buffers (albedo, depth, metallic, normal, roughness), and the model
renders them into RGB frames in a style set by a text prompt. It streams
autoregressively with 4 denoising steps, at about 30 FPS on one NVIDIA H200.

These are the weights behind the [live demo](https://alaya-renderer-flash.alayalab.ai),
fine-tuned for SuperTuxKart.

- **Paper:** [Generative World Renderer at the Speed of Play](https://arxiv.org/abs/2607.18703)
- **Code:** [github.com/AlayaLab/AlayaRenderer-Flash](https://github.com/AlayaLab/AlayaRenderer-Flash)
- **Base model:** [Wan-AI/Wan2.1-T2V-1.3B-Diffusers](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers)
  (supplies the text encoder, tokenizer and VAE config; download it separately)

## Files

| File | Size | Description |
|---|---|---|
| `transformer/` | 3.1 GB | 1.3B `HeliosTransformer3DModel` (bf16): 96 input channels (16 RGB + 5×16 G-buffer latents), text-sink, multi-term memory patches |
| `transformer_partial.safetensors` | 0.3 GB | G-buffer patch embedding, short/mid/long memory patchifiers, output scale-shift table, text-sink weights (loaded on top of `transformer/`) |
| `gbuf_small_encoder.safetensors` | 37 MB | Grouped distilled G-buffer encoder (5 branches, hidden dim 128) that replaces the Wan VAE encoder |
| `tiny_decoder/taew2_1_ft.safetensors` | 45 MB | Fine-tuned TAEW2.1 stateful tiny decoder that replaces the Wan VAE decoder |

## Usage

```bash
git clone https://github.com/AlayaLab/AlayaRenderer-Flash && cd AlayaRenderer-Flash
uv sync
uv run python examples/fetch_samples.py
uv run alayarender-flash render samples/window-00.npy samples/window-01.npy \
    --prompt "heavy snow at night, warm street lights" -o snow.mp4
```

The CLI downloads these weights and the Wan 2.1 base model on first use.
`samples/` has two recorded SuperTuxKart G-buffer windows
(`uint8 (16, 5, 448, 832, 3)`, channel order albedo, depth, metallic, normal,
roughness). For the interactive SuperTuxKart demo, see the repository README.

## Model details

- **Architecture:** Wan 2.1 1.3B DiT extended with the Helios autoregressive
  design: unified history injection, multi-term memory patchification with a
  long/mid/short history of 8/2/1 latents, first-frame anchor and text-sink
  prompt anchoring. G-buffer latents are concatenated with the noisy RGB latent
  before patch embedding.
- **Inference:** 832×448, latent window 4 (13 pixel frames to start, then 16
  per window), 4 Euler steps, guidance scale 1.0 (CFG is distilled into the model).
- **Training:** fine-tuned on SuperTuxKart captures from the AlayaRenderer-Flash
  base renderer with autoregressive training, CFG distillation, progressive
  4-step distillation and Mean Flow Distillation (see the paper). This is
  checkpoint 600 of the final stage. Training code is not part of this release.

## Intended use and limitations

- This model is for research on G-buffer-conditioned neural rendering and
  interactive generative worlds. It is tuned for SuperTuxKart G-buffers at
  832×448, and quality drops on other engines, other G-buffer conventions or
  other resolutions.
- Prompt control is limited: the distilled student runs at guidance scale 1.0,
  and some styles barely change the output.
- In long rollouts it can drift in color and lighting. Reset the history when
  the scene or prompt changes, as the demo does.
- Real-time rates need a Hopper-class GPU. Consumer GPUs run, but slower.

## Citation

```bibtex
@article{lin2026alayarendererflash,
  title   = {Generative World Renderer at the Speed of Play},
  author  = {Lin, Guixu and Huang, Zheng-Hui and Yang, Siqi and Yang, Ming-Hsuan and Zhang, Kaipeng and Wang, Zhixiang},
  journal = {arXiv preprint arXiv:2607.18703},
  year    = {2026}
}
```