| --- |
| license: apache-2.0 |
| base_model: Wan-AI/Wan2.1-T2V-1.3B-Diffusers |
| library_name: diffusers |
| pipeline_tag: video-to-video |
| tags: |
| - g-buffer |
| - neural-rendering |
| - real-time |
| - autoregressive |
| - video-diffusion |
| - supertuxkart |
| --- |
| |
| # AlayaRenderer-Flash |
|
|
| AlayaRenderer-Flash is a real-time generative world renderer. A game engine |
| exports G-buffers (albedo, depth, metallic, normal, roughness), and the model |
| renders them into RGB frames in a style set by a text prompt. It streams |
| autoregressively with 4 denoising steps, at about 30 FPS on one NVIDIA H200. |
|
|
| These are the weights behind the [live demo](https://alaya-renderer-flash.alayalab.ai), |
| fine-tuned for SuperTuxKart. |
|
|
| - **Paper:** [Generative World Renderer at the Speed of Play](https://arxiv.org/abs/2607.18703) |
| - **Code:** [github.com/AlayaLab/AlayaRenderer-Flash](https://github.com/AlayaLab/AlayaRenderer-Flash) |
| - **Base model:** [Wan-AI/Wan2.1-T2V-1.3B-Diffusers](https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B-Diffusers) |
| (supplies the text encoder, tokenizer and VAE config; download it separately) |
|
|
| ## Files |
|
|
| | File | Size | Description | |
| |---|---|---| |
| | `transformer/` | 3.1 GB | 1.3B `HeliosTransformer3DModel` (bf16): 96 input channels (16 RGB + 5×16 G-buffer latents), text-sink, multi-term memory patches | |
| | `transformer_partial.safetensors` | 0.3 GB | G-buffer patch embedding, short/mid/long memory patchifiers, output scale-shift table, text-sink weights (loaded on top of `transformer/`) | |
| | `gbuf_small_encoder.safetensors` | 37 MB | Grouped distilled G-buffer encoder (5 branches, hidden dim 128) that replaces the Wan VAE encoder | |
| | `tiny_decoder/taew2_1_ft.safetensors` | 45 MB | Fine-tuned TAEW2.1 stateful tiny decoder that replaces the Wan VAE decoder | |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/AlayaLab/AlayaRenderer-Flash && cd AlayaRenderer-Flash |
| uv sync |
| uv run python examples/fetch_samples.py |
| uv run alayarender-flash render samples/window-00.npy samples/window-01.npy \ |
| --prompt "heavy snow at night, warm street lights" -o snow.mp4 |
| ``` |
|
|
| The CLI downloads these weights and the Wan 2.1 base model on first use. |
| `samples/` has two recorded SuperTuxKart G-buffer windows |
| (`uint8 (16, 5, 448, 832, 3)`, channel order albedo, depth, metallic, normal, |
| roughness). For the interactive SuperTuxKart demo, see the repository README. |
|
|
| ## Model details |
|
|
| - **Architecture:** Wan 2.1 1.3B DiT extended with the Helios autoregressive |
| design: unified history injection, multi-term memory patchification with a |
| long/mid/short history of 8/2/1 latents, first-frame anchor and text-sink |
| prompt anchoring. G-buffer latents are concatenated with the noisy RGB latent |
| before patch embedding. |
| - **Inference:** 832×448, latent window 4 (13 pixel frames to start, then 16 |
| per window), 4 Euler steps, guidance scale 1.0 (CFG is distilled into the model). |
| - **Training:** fine-tuned on SuperTuxKart captures from the AlayaRenderer-Flash |
| base renderer with autoregressive training, CFG distillation, progressive |
| 4-step distillation and Mean Flow Distillation (see the paper). This is |
| checkpoint 600 of the final stage. Training code is not part of this release. |
|
|
| ## Intended use and limitations |
|
|
| - This model is for research on G-buffer-conditioned neural rendering and |
| interactive generative worlds. It is tuned for SuperTuxKart G-buffers at |
| 832×448, and quality drops on other engines, other G-buffer conventions or |
| other resolutions. |
| - Prompt control is limited: the distilled student runs at guidance scale 1.0, |
| and some styles barely change the output. |
| - In long rollouts it can drift in color and lighting. Reset the history when |
| the scene or prompt changes, as the demo does. |
| - Real-time rates need a Hopper-class GPU. Consumer GPUs run, but slower. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{lin2026alayarendererflash, |
| title = {Generative World Renderer at the Speed of Play}, |
| author = {Lin, Guixu and Huang, Zheng-Hui and Yang, Siqi and Yang, Ming-Hsuan and Zhang, Kaipeng and Wang, Zhixiang}, |
| journal = {arXiv preprint arXiv:2607.18703}, |
| year = {2026} |
| } |
| ``` |
|
|
