| --- |
| base_model: nvidia/Cosmos3-Nano |
| license: other |
| license_name: openmdw1.1-license |
| license_link: https://openmdw.ai/license/1-1/ |
| pipeline_tag: image-to-video |
| tags: |
| - world-model |
| - neural-rendering |
| - video-generation |
| - autoregressive |
| --- |
| |
| # AgentGarten renderer |
|
|
| The real-time neural renderer of [AgentGarten: Code Worlds for Evolving Agents](https://huggingface.co/papers/2610.12374). |
| A code world (a simulator or a game engine) keeps the state and runs the rules; this model turns the depth and surface |
| normals it exports into the observations an agent sees, block by block. |
|
|
| - Paper: https://huggingface.co/papers/2610.12374 |
| - Blog: https://mirros.ai/blog/worlds-for-evolving-agents |
| - Technical report: https://mirros.ai/report/agent-garten.pdf |
| - Project page: https://mirros-lab.github.io/agent-garten |
| - Code: https://github.com/MirroS-Lab/AgentGarten |
|
|
| ## What is in this repository |
|
|
| | File | Contents | |
| | --- | --- | |
| | `student.safetensors` | The few-step autoregressive generation tower (7.0 B parameters, BF16), distilled with Adversarial Forcing | |
| | `und.safetensors` | The frozen text tower of Cosmos3-Nano, unchanged | |
| | `text_tokenizer/` | The Cosmos3-Nano text tokenizer, unchanged | |
| | `config.yaml`, `manifest.json` | Network configuration and the file list the loader reads | |
|
|
| The model starts from [Cosmos3-Nano](https://huggingface.co/nvidia/Cosmos3-Nano). It was adapted to depth and normal |
| conditions, trained as a block-causal autoregressive model, and distilled into a four-step student. Video is encoded |
| and decoded with the [Wan 2.2 VAE](https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B), which is not included here. |
|
|
| ## Use |
|
|
| Install the [AgentGarten code](https://github.com/MirroS-Lab/AgentGarten), then: |
|
|
| ```bash |
| hf download MirroS-Lab/AgentGarten-renderer --local-dir weights/AgentGarten-renderer |
| hf download Wan-AI/Wan2.2-TI2V-5B Wan2.2_VAE.pth --local-dir weights/Wan2.2-TI2V-5B |
| ``` |
|
|
| ```python |
| from wm.inference.cosmos3 import inspect_artifact, load_artifact |
| from wm.inference.serving import prepare_serving |
| from wm.networks.cosmos3.streaming import Cosmos3Stream, StreamPolicy |
| |
| artifact = load_artifact(inspect_artifact("weights/AgentGarten-renderer")) |
| prepare_serving(artifact.student) |
| stream = Cosmos3Stream(artifact.student, StreamPolicy()) |
| ``` |
|
|
| The repository's README shows the streaming loop: each block of four latent frames is denoised in four steps and then |
| committed to the key-value cache, so the conditions of the next block can depend on what was just generated. |
|
|
| ## License |
|
|
| The weights derive from Cosmos3-Nano and are released under the same |
| [OpenMDW License Agreement, version 1.1](https://openmdw.ai/license/1-1/). The text tower and the tokenizer are |
| NVIDIA's files, redistributed unchanged under that license. |
