nielsr HF Staff commited on
Commit
75b4d2d
·
verified ·
1 Parent(s): 778f52e

Replace incorrect model card with Genie Sim 3.0 card

Browse files

The existing model card describes a different project (NVIDIA DreamZero). This PR replaces it with an accurate card for Genie Sim 3.0, adds the `pipeline_tag: robotics`, sets the license to `mpl-2.0`, and links to the paper and code repository.

Files changed (1) hide show
  1. README.md +14 -270
README.md CHANGED
@@ -1,274 +1,18 @@
1
- # NVIDIA DreamZero: World Action Models Are Zero-Shot Policies
2
- A research project from [NVIDIA GEAR Lab](https://research.nvidia.com/labs/gear/).
 
 
3
 
4
- [![NVIDIA](https://img.shields.io/badge/NVIDIA-76B900?style=flat&logo=nvidia&logoColor=white)](https://www.nvidia.com) [![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE) [![arXiv](https://img.shields.io/badge/arXiv-2602.15922-b31b1b.svg)](https://arxiv.org/abs/2602.15922)
5
 
6
- [[Project Page](https://dreamzero0.github.io/)] [[Paper](https://arxiv.org/abs/2602.15922)]
7
 
8
- DreamZero is a World Action Model that jointly predicts actions and videos, achieving strong zero-shot performance on unseen tasks. This release package contains everything needed to load a pretrained DreamZero model and run distributed inference via a WebSocket server.
 
 
 
 
9
 
10
- ## News
11
-
12
- - **02/27:** DreamZero is **#1 on both [MolmoSpaces]([https://huggingface.co/spaces/ai2-adapt/MolmoSpaces](https://molmospaces.allen.ai/leaderboard)) and [RoboArena]([https://robo-arena.github.io/](https://robo-arena.github.io/leaderboard))**! DreamZero-DROID is trained *from scratch* using only the DROID dataset — no pretraining on large-scale robot data, unlike competing VLAs. This demonstrates the strength of video-model backbones for generalist robot policies (VAMs/WAMs).
13
- - **02/27:** Released **DreamZero-AgiBot checkpoint** and **post-training code** for efficient few-shot adaptation. Post-train on just ~30 minutes of play data for your specific robot, and see the robot do basic language following and pick-and-place (see YAM experiments in our paper for more detail).
14
- - **02/20:** Released the **full training codebase, preprocessed dataset, and guide for new embodiments** to replicate the DreamZero-DROID checkpoint and train on your own robot. See [Adding a New Embodiment to DreamZero](docs/DATASET_TO_GEAR_AND_TRAIN.md) for a step-by-step walkthrough.
15
-
16
- ## Features
17
-
18
- **Available Now**
19
- - Pretrained DreamZero-DROID model checkpoint [[Huggingface](https://huggingface.co/GEAR-Dreams/DreamZero-DROID)]
20
- - Pretrained DreamZero-AgiBot checkpoint (for post-training on new embodiments) [[Huggingface](https://huggingface.co/GEAR-Dreams/DreamZero-AgiBot)]
21
- - Distributed WebSocket inference server (GB200, H100)
22
- - DiT caching for optimized inference (~0.6s on GB200, ~3s on H100)
23
- - DROID simulation evaluation support
24
- - [RoboArena](https://robo-arena.github.io/) integration (DROID real)
25
- - Video generation and saving (MP4)
26
- - LoRA and full fine-tuning training scripts
27
- - Training on new embodiments (AgiBot, YAM) — see [guide](docs/DATASET_TO_GEAR_AND_TRAIN.md)
28
-
29
- **Coming Soon**
30
- - [PolaRiS](https://polaris-evals.github.io/) simulation environment support
31
- - [Genie 3.0](https://arxiv.org/abs/2601.02078) sim environment support for DreamZero-AgiBot
32
-
33
- ## Testing Out DreamZero in Simulation with API
34
- We provide an inference script that directly evaluates a hosted DreamZero-DROID policy on [`sim_evals`](https://github.com/arhanjain/sim-evals). To test out the policy, first request access to the API via this form [link](https://forms.gle/zCj5zjDvHsoeuMXU7). Then, follow these instructions to install [`sim_evals`](https://github.com/arhanjain/sim-evals) and launch evaluation.
35
-
36
- ```bash
37
- # Clone repository
38
- git clone --recurse-submodules https://github.com/arhanjain/sim-evals.git
39
- cd sim-evals
40
-
41
- # Install uv
42
- curl -LsSf https://astral.sh/uv/install.sh | sh
43
-
44
- # Activate uv environment
45
- uv sync
46
- source .venv/bin/activate
47
-
48
- # [Optional] update pytorch versions
49
- pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 --index-url https://download.pytorch.org/whl/cu129
50
-
51
- # Download assets (may need to export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN> first)
52
- uvx hf download owhan/DROID-sim-environments --repo-type dataset --local-dir assets
53
-
54
- # Run eval script
55
- cd ..
56
- python eval_utils/run_sim_eval.py --host <API_HOST> --port <API_PORT>
57
- ```
58
-
59
- The outputs are saved in `runs` directory.
60
-
61
-
62
- ## Quick Start
63
-
64
- ### Prerequisites
65
-
66
- - **Python**: 3.11
67
- - **Hardware**: Multi-GPU setup (tested on GB200, H100)
68
- - Minimum: 2 GPUs for distributed inference
69
- - **CUDA**: Compatible GPU with CUDA 12.9+
70
-
71
- ### Installation
72
-
73
- 1. **Create conda environment:**
74
- ```bash
75
- conda create -n dreamzero python=3.11
76
- conda activate dreamzero
77
- ```
78
-
79
- 2. **Install dependencies (PyTorch 2.8+ with CUDA 12.9+):**
80
- ```bash
81
- pip install -e . --extra-index-url https://download.pytorch.org/whl/cu129
82
- ```
83
-
84
- 3. **Install flash attention:**
85
- ```bash
86
- MAX_JOBS=8 pip install --no-build-isolation flash-attn
87
- ```
88
-
89
- 4. **[GB200 ONLY, SKIP FOR H100] Install Transformer Engine:**
90
- ```bash
91
- pip install --no-build-isolation transformer_engine[pytorch]
92
- ```
93
-
94
- 5. **[GB200 ONLY FOR TENSORRT, SKIP FOR H100] Install Tensorrt:**
95
- ```bash
96
- pip install tensorrt==10.13.2.6 tensorrt_cu13==10.13.2.6 tensorrt_cu13_libs==10.13.2.6 tensorrt_cu13_bindings==10.13.2.6 --no-deps
97
- pip install transformer_engine==2.10.0 transformer_engine_cu12==2.10.0 transformer_engine_torch==2.10.0
98
- ```
99
-
100
- ## Downloading Pretrained Checkpoints
101
-
102
- ### DreamZero-DROID (for inference)
103
-
104
- We release a 14B pretrained DROID checkpoint on [Huggingface](https://huggingface.co/GEAR-Dreams/DreamZero-DROID). To download the checkpoint, run
105
-
106
- ```bash
107
- hf download GEAR-Dreams/DreamZero-DROID --repo-type model --local-dir <path/to/checkpoint>
108
- ```
109
-
110
- ### DreamZero-AgiBot (for fine-tuning on new embodiments)
111
-
112
- To fine-tune DreamZero on a new embodiment (e.g. YAM, AgiBot), download the pretrained [DreamZero-AgiBot](https://huggingface.co/GEAR-Dreams/DreamZero-AgiBot) checkpoint (~45GB) to `./checkpoints/DreamZero-AgiBot`:
113
-
114
- ```bash
115
- git clone https://huggingface.co/GEAR-Dreams/DreamZero-AgiBot ./checkpoints/DreamZero-AgiBot
116
- ```
117
-
118
- Or with the Hugging Face CLI:
119
-
120
- ```bash
121
- hf download GEAR-Dreams/DreamZero-AgiBot --repo-type model --local-dir ./checkpoints/DreamZero-AgiBot
122
- ```
123
-
124
- The YAM and AgiBot training scripts use `pretrained_model_path=./checkpoints/DreamZero-AgiBot` by default. See the [new embodiment guide](docs/DATASET_TO_GEAR_AND_TRAIN.md) for usage.
125
-
126
- ## Running the Inference Server
127
-
128
- ### Command Overview
129
-
130
- The inference server uses PyTorch distributed training utilities to parallelize the model across multiple GPUs:
131
-
132
- ```bash
133
- CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.run --standalone --nproc_per_node=2 socket_test_optimized_AR.py --port 5000 --enable-dit-cache --model-path <path/to/checkpoint>
134
- ```
135
-
136
- (Optional only for GB200) Tensorrt enables faster generation
137
- ```bash
138
- export LOAD_TRT_ENGINE=<path/to/checkpoint>/tensorrt/wan/WanModel_nvfp4.trt
139
- export DYNAMIC_CACHE_SCHEDULE=true
140
- CUDA_VISIBLE_DEVICES=0,1 python -m torch.distributed.run --standalone --nproc_per_node=2 /mnt/aws-lfs-02/shared/seonghyeony/dreamzero/socket_test_optimized_AR.py --port 8000 --enable-dit-cache --model-path <path/to/checkpoint>
141
- ```
142
- To verify the server is working, run a test client. The first few inferences will take a few minutes to warm up. After warming up, inference takes ~0.6s on GB200 and ~3s on H100.
143
-
144
- ```
145
- python test_client_AR.py --port 5000
146
- ```
147
-
148
- ### Command-line Arguments
149
-
150
- - `--port`: Port number for the WebSocket server (default: 8000)
151
- - `--model-path`: Path to the pretrained model checkpoint directory
152
- - `--enable-dit-cache`: Enable caching in DiT layers for faster inference (recommended)
153
- - `--max-chunk-size`: Override max_chunk_size for inference (optional)
154
- - `--timeout-seconds`: Server timeout in seconds (default: 50000)
155
- - `--index`: Index for output directory naming (default: 0)
156
-
157
-
158
- ### Output
159
-
160
- The server saves:
161
- - **Videos**: Generated video predictions as MP4 files in `{model_path}/real_world_eval_gen_{date}_{index}/{checkpoint_name}/`
162
- - **Input observations**: Saved per message in `{output_dir}/inputs/{msg_index}_{timestamp}/`
163
-
164
-
165
- ## Training
166
-
167
- > **Training on a new embodiment?** See [Adding a New Embodiment to DreamZero](docs/DATASET_TO_GEAR_AND_TRAIN.md) for a complete guide on converting your dataset, configuring modalities, and launching training. <em>Make sure to align the 3 camera view order to ensure positive transfer.</em>
168
-
169
- ### Downloading Pretrained Base Model Weights
170
-
171
- DreamZero is built on top of [Wan2.1-I2V-14B-480P](https://huggingface.co/Wan-AI/Wan2.1-I2V-14B-480P) and uses the [umt5-xxl](https://huggingface.co/google/umt5-xxl) tokenizer. Download both before training:
172
-
173
- ```bash
174
- pip install "huggingface_hub[cli]"
175
-
176
- # You may need to set your HuggingFace token:
177
- # export HF_TOKEN=<YOUR_HUGGINGFACE_TOKEN>
178
-
179
- # Download Wan2.1 model weights (~28GB)
180
- hf download Wan-AI/Wan2.1-I2V-14B-480P --local-dir ./checkpoints/Wan2.1-I2V-14B-480P
181
-
182
- # Download umt5-xxl tokenizer
183
- hf download google/umt5-xxl --local-dir ./checkpoints/umt5-xxl
184
- ```
185
-
186
- > **Note:** The training script will auto-download these if they are not found at the configured paths, but pre-downloading is recommended to avoid delays at launch.
187
-
188
- ### DROID Dataset
189
-
190
- We release the preprocessed DROID dataset used to train DreamZero on HuggingFace: [GEAR-Dreams/DreamZero-DROID-Data](https://huggingface.co/datasets/GEAR-Dreams/DreamZero-DROID-Data).
191
-
192
- This dataset is derived from the [DROID 1.0.1](https://droid-dataset.github.io/) dataset with the following modifications:
193
- - Converted from RLDS/TFDS format to [LeRobot](https://github.com/huggingface/lerobot) v2.0 format
194
- - Idle frames removed using [Physical Intelligence's idle frame detector](https://github.com/Physical-Intelligence/openpi/blob/main/examples/droid/README_train.md#data-filtering) (`droid_sample_ranges_v1_0_1.json`)
195
- - Episodes without language annotations are filtered out
196
- - Successful episodes only (episodes with non-zero reward)
197
- - 3 camera views: `exterior_image_1_left`, `exterior_image_2_left`, `wrist_image_left`
198
-
199
- **To download the preprocessed dataset (~131GB):**
200
-
201
- ```bash
202
- huggingface-cli download GEAR-Dreams/DreamZero-DROID-Data --repo-type dataset --local-dir ./data/droid_lerobot
203
- ```
204
-
205
- If you want to reproduce the dataset conversion from raw DROID 1.0.1 yourself (or modify the filtering), see [docs/DROID_CONVERSION.md](docs/DROID_CONVERSION.md).
206
-
207
- ### Running Training
208
-
209
- ```bash
210
- # Configure paths (override defaults as needed)
211
- export DROID_DATA_ROOT="./data/droid_lerobot"
212
- export OUTPUT_DIR="./checkpoints/dreamzero_droid"
213
- export NUM_GPUS=4
214
-
215
- # Point to your downloaded model weights (if not using default paths)
216
- export WAN_CKPT_DIR="./checkpoints/Wan2.1-I2V-14B-480P"
217
- export TOKENIZER_DIR="./checkpoints/umt5-xxl"
218
-
219
- # Launch training
220
- bash scripts/train/droid_training.sh
221
- ```
222
-
223
- **Using Wan2.2-TI2V-5B backbone (5B params, lower VRAM):** To train with the smaller Wan2.2-TI2V-5B model instead of Wan2.1-I2V-14B, see [docs/WAN22_BACKBONE.md](docs/WAN22_BACKBONE.md) and run `bash scripts/train/droid_training_wan22.sh`.
224
-
225
- ### Training Configuration
226
-
227
- The training script uses Hydra for configuration and DeepSpeed ZeRO Stage 2 for distributed training. Key defaults:
228
-
229
- | Parameter | Default | Description |
230
- |---|---|---|
231
- | `NUM_GPUS` | 4 | Number of GPUs |
232
- | `per_device_train_batch_size` | 1 | Batch size per GPU |
233
- | `learning_rate` | 1e-5 | Learning rate |
234
- | `max_steps` | 10 | Max training steps (increase for full training) |
235
- | `warmup_ratio` | 0.05 | Warmup ratio |
236
- | `weight_decay` | 1e-5 | Weight decay |
237
- | `image_resolution_width` | 320 | Image width |
238
- | `image_resolution_height` | 176 | Image height |
239
- | `num_frames` | 33 | Number of video frames |
240
- | `action_horizon` | 24 | Action prediction horizon |
241
- | `save_lora_only` | true | Only save LoRA weights |
242
- | `bf16` | true | Use bfloat16 precision |
243
-
244
- > **Note:** `max_steps=10` is set for a quick sanity check. For full training, increase this to your desired number of steps and configure `save_steps` / `save_strategy` accordingly.
245
-
246
-
247
- ## Citation
248
-
249
- If you use DreamZero in your research, please cite:
250
-
251
- ```bibtex
252
- @misc{ye2026worldactionmodelszeroshot,
253
- title={World Action Models are Zero-shot Policies},
254
- author={Seonghyeon Ye and Yunhao Ge and Kaiyuan Zheng and Shenyuan Gao and Sihyun Yu and George Kurian and Suneel Indupuru and You Liang Tan and Chuning Zhu and Jiannan Xiang and Ayaan Malik and Kyungmin Lee and William Liang and Nadun Ranawaka and Jiasheng Gu and Yinzhen Xu and Guanzhi Wang and Fengyuan Hu and Avnish Narayan and Johan Bjorck and Jing Wang and Gwanghyun Kim and Dantong Niu and Ruijie Zheng and Yuqi Xie and Jimmy Wu and Qi Wang and Ryan Julian and Danfei Xu and Yilun Du and Yevgen Chebotar and Scott Reed and Jan Kautz and Yuke Zhu and Linxi "Jim" Fan and Joel Jang},
255
- year={2026},
256
- eprint={2602.15922},
257
- archivePrefix={arXiv},
258
- primaryClass={cs.RO},
259
- url={https://arxiv.org/abs/2602.15922},
260
- }
261
- ```
262
-
263
- ## License
264
-
265
- This project is licensed under the [Apache License 2.0](LICENSE).
266
-
267
- ## Support
268
-
269
- For issues and questions:
270
- - Check the troubleshooting section above
271
- - Review server logs for detailed error messages
272
- - Verify your checkpoint is compatible with this release
273
-
274
- [![Star History Chart](https://api.star-history.com/svg?repos=dreamzero0/dreamzero&type=Date)](https://star-history.com/#dreamzero0/dreamzero&Date)
 
1
+ ---
2
+ license: mpl-2.0
3
+ pipeline_tag: robotics
4
+ ---
5
 
6
+ # Genie Sim 3.0: A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot
7
 
8
+ Genie Sim is a unified simulation platform for robotic manipulation developed by AgiBot. It provides a complete toolchain for environment reconstruction, scene generalization, data collection, and automated evaluation.
9
 
10
+ Key features:
11
+ - **LLM-driven scene generation** from natural language instructions
12
+ - **VLM-based automated evaluation** across 200+ tasks and 100,000+ scenarios
13
+ - **Open-source synthetic dataset** with over 10,000 hours of data
14
+ - **High-fidelity simulation** with demonstrated zero-shot sim-to-real transfer
15
 
16
+ **Paper:** [Genie Sim 3.0 : A High-Fidelity Comprehensive Simulation Platform for Humanoid Robot](https://huggingface.co/papers/2601.02078)
17
+ **Code:** [github.com/AgibotTech/genie_sim](https://github.com/AgibotTech/genie_sim)
18
+ **Project page:** [agibot-world.com/genie-sim](http://agibot-world.com/genie-sim)