anygame-renderer / README.md
multimodalart's picture
multimodalart HF Staff
Upload folder using huggingface_hub
d932406 verified
|
Raw History Blame Contribute Delete
4.49 kB
---
title: AnyGameRenderer
emoji: 🎮
colorFrom: blue
colorTo: indigo
sdk: gradio
sdk_version: 6.30.0
app_file: app.py
pinned: false
short_description: Blockout gameplay clips re-rendered as polished game video
python_version: "3.12"
startup_duration_timeout: 1h
models:
- AlayaLab/AnyGameRenderer
- multimodalart/MiniMax-H3-Pruned
- MiniMaxAI/MiniMax-H3
---
# AnyGameRenderer
A demo of [`AlayaLab/AnyGameRenderer`](https://huggingface.co/AlayaLab/AnyGameRenderer), the entity-consistent
generative game renderer from [*AnyGameRenderer: Entity-Consistent Generative Rendering Across Games and an
Entity-Level Benchmark*](https://arxiv.org/pdf/2610.31466). Hand it the crude "programmer art" a game agent produces
— a pale-gray blockout with one flat, saturated proxy color per entity class — plus the caption the paper trains on,
and it returns the same gameplay re-rendered as a polished game clip: frame-aligned with the input, group counts and
appearance timing intact, with a jointly generated soundtrack. One checkpoint covers shooters, kart racers, city
driving, survivors-likes, turn-based battles and more.
## The request
Two inputs, both required:
| Slot | What it is |
|---|---|
| **Blockout clip** (`<Video 1>`) | an mp4 at ~24 fps, 2–15 s. 1344×768 recommended. Pale-gray geometry; one flat proxy color (red, orange, yellow, yellow-green, green, cyan, blue, purple, magenta) per character / weapon / vehicle class. No HUD, text or effects. |
| **Caption** | the six-section caption the LoRA was trained on (subject_definitions, summary, retention_analysis, detailed_description, overall_soundscape, non_diegetic_music). The **caption builder** in the accordion builds it from a short spec, exactly as the paper's `make_caption.py` fills the training layout. |
The three bundled examples are the paper's own — the WW1 shooter, the third-person egg heist and the city-driving
clips its repo ships, with the captions built from their `spec.json` files.
## How it is deployed
MiniMax-H3 is 195.9 GiB in bfloat16 and a ZeroGPU Space is evicted at 150 GB of storage, so the deployment is
split the same way as its sibling `hugging-apps/minimax-h3-character-swap-lora`: this Space is the **denoising
half** of `ref2va` (the `transformer_ref` partition and the two autoencoders), and the 62 GiB Qwen3-VL conditioner
runs in [`multimodalart/qwen3vl-conditioner`](https://huggingface.co/spaces/multimodalart/qwen3vl-conditioner),
called over the gradio API per request. `prompt_embeds` + `text_token_tags` is the whole wire format.
The DiT is [`multimodalart/MiniMax-H3-Pruned`](https://huggingface.co/multimodalart/MiniMax-H3-Pruned)'s
`transformer_ref` (37.5 GiB): the released partition with its AdaLN input projections folded onto their reachable
rank. The AnyGameRenderer LoRA — rank 128, alpha 128, on `attn.qkv_proj`, `attn.out_proj`, `mlp.fc1`, `mlp.fc2` of
every transformer block — is AdaLN-free by construction, so it applies verbatim; it is attached as a runtime PEFT
adapter so the strength slider stays a per-request knob and **0.0 is the zero-shot base model**, the paper's own
baseline. Loading goes through diffusers 0.41's `MiniMaxH3LoraLoaderMixin`, whose
`_convert_non_diffusers_minimax_h3_lora_to_diffusers` handles this file's flattened musubi-tuner
`lora_unet_blocks_…` names, folds the `.alpha` scalars in, splits the fused QKV and swaps the SwiGLU gate/value
halves — the same transforms the paper's `infer.py` applies through musubi-tuner's merge.
Defaults are the paper's own inference settings: 1344×768, 124 frames (the training bucket), 20 steps, seed 42,
strength 1.0. The GPU reservation is a fitted cost model over the packed sequence, evaluated per request.
## Notes
- Resolution is fixed to 1344×768 — what the LoRA was trained on. Length is snapped to `17n+5` frames; clips longer
than 15 s are refused by the base model.
- Generation is deterministic for a given seed; different seeds change appearance details, not layout.
- The soundtrack is *generated*, not carried over from the blockout (which has no audio).
## Example assets and license
The three example clips and their captions are from [`AlayaLab/AnyGameRenderer`](https://github.com/AlayaLab/AnyGameRenderer)
(Apache-2.0), the paper's own repository. The adapter is distributed under the
[MiniMax H3 Community License Agreement](https://huggingface.co/MiniMaxAI/MiniMax-H3/blob/main/LICENSE) — read the
upstream terms; nothing here extends them.