Instructions to use tz2026/ReWorld with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Diffusers
How to use tz2026/ReWorld with Diffusers:
pip install -U diffusers transformers accelerate
import torch from diffusers import DiffusionPipeline # switch to "mps" for apple devices pipe = DiffusionPipeline.from_pretrained("tz2026/ReWorld", dtype=torch.bfloat16, device_map="cuda") prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k" image = pipe(prompt).images[0] - Notebooks
- Google Colab
- Kaggle
Update README.md
Browse files
README.md
CHANGED
|
@@ -1,19 +1,16 @@
|
|
| 1 |
---
|
| 2 |
-
license: apache-2.0
|
| 3 |
tags:
|
| 4 |
-
|
| 5 |
-
|
| 6 |
-
|
| 7 |
-
|
| 8 |
-
- diffusion
|
| 9 |
---
|
| 10 |
|
| 11 |
<div align="center">
|
| 12 |
|
| 13 |
-
|
| 14 |
|
| 15 |
-
**The first representation learning framework for autonomous-driving World Action Models
|
| 16 |
-
explicitly optimizing the latent world-to-action pathway.**
|
| 17 |
|
| 18 |
[](https://arxiv.org/abs/2606.27504)
|
| 19 |
[](https://xiaomi-research.github.io/ReWorld/)
|
|
@@ -33,14 +30,14 @@ explicitly optimizing the latent world-to-action pathway.**
|
|
| 33 |
|
| 34 |
---
|
| 35 |
|
| 36 |
-
##
|
| 37 |
|
| 38 |
- **Better futures** β FVD **81.3 β 61.9** (β23.9%) on nuScenes video generation, with self-guided sampling enabled for free by intermediate supervision.
|
| 39 |
- **Safer plans** β closed-loop PDMS **89.1 β 90.4** on NAVSIM *Navtest*, with best-in-class NC / DAC / TTC among world-model planners β no RL, no test-time scoring.
|
| 40 |
- **Stronger representations** β frozen linear probe on UCF-101 action recognition **68.3% β 80.2%** (+11.9 pts over the DriveLaW baseline).
|
| 41 |
- **Nearly free** β no external encoders, no teacher models, only **+0.3%** per-step Video DiT training cost, and roughly **2Γ faster** convergence from scratch.
|
| 42 |
|
| 43 |
-
##
|
| 44 |
|
| 45 |
World Action Models (WAMs) couple future environment prediction with action generation, yet standard training supervises only the output ends of the generation and planning modules. The intermediate representations that carry world knowledge are shaped only indirectly, as byproducts of fitting these outputs. We call this the **representation bottleneck of WAMs**: the world-to-action pathway is not explicitly optimized to be future-predictive, cross-modally grounded, or sensitive to closed-loop behavior quality.
|
| 46 |
|
|
@@ -58,7 +55,7 @@ All supervision is constructed entirely from the WAM's own generation targets, a
|
|
| 58 |
<img src="assets/reworld_self_guidance.png" width="100%" alt="Self-guided sampling and faster convergence">
|
| 59 |
</div>
|
| 60 |
|
| 61 |
-
##
|
| 62 |
|
| 63 |
<details open><summary><b>Video generation on nuScenes</b> (val)</summary>
|
| 64 |
|
|
@@ -127,7 +124,7 @@ Driving-domain pretraining alone barely moves the probe (+1.5 over LTX-Video); t
|
|
| 127 |
<br><i>NAVSIM Navtest rollouts (red: ReWorld prediction, green: expert) β straight, left turn, right turn, intersection.</i>
|
| 128 |
</div>
|
| 129 |
|
| 130 |
-
##
|
| 131 |
|
| 132 |
| Folder | File | Role |
|
| 133 |
|:---|:---|:---|
|
|
@@ -148,7 +145,7 @@ hf download tz2026/ReWorld
|
|
| 148 |
|
| 149 |
Please download the **whole repo** (not a single `.safetensors` file). Hugging Face only increments download stats when `config.json` is requested, which `snapshot_download` / `hf download` always fetch.
|
| 150 |
|
| 151 |
-
##
|
| 152 |
|
| 153 |
### Installation
|
| 154 |
|
|
@@ -215,11 +212,11 @@ sh scripts/evaluation/run_videodrive_agent_pdm_score_evaluation.sh
|
|
| 215 |
|
| 216 |
> Optional base imitation before Stage 2: `scripts/training/run_videodrive_train.sh` with [`video_model_train_base.yaml`](DriveLaW-Act/navsim/agents/videodrive/configs/ltx_model/video_model_train_base.yaml), or start from the π€ DriveLaW checkpoint.
|
| 217 |
|
| 218 |
-
##
|
| 219 |
|
| 220 |
-
ReWorld builds on [DriveLaW](https://arxiv.org/abs/2512.23421), [NAVSIM](https://github.com/autonomousvision/navsim), [LTX-Video](https://github.com/Lightricks/LTX-Video), [Diffusers](https://github.com/huggingface/diffusers), and the hard-negative protocol of [BeyondDrive](https://github.com/wjl2244/BeyondDrive). Stage 1 uses intermediate supervision inspired by [Internal Guidance](https://
|
| 221 |
|
| 222 |
-
##
|
| 223 |
|
| 224 |
```bibtex
|
| 225 |
@article{xia2026reworld,
|
|
@@ -232,4 +229,4 @@ ReWorld builds on [DriveLaW](https://arxiv.org/abs/2512.23421), [NAVSIM](https:/
|
|
| 232 |
|
| 233 |
## :mailbox: Contact
|
| 234 |
|
| 235 |
-
Tianze Xia β xiatianze@hust.edu.cn Β· Xinggang Wang β xgwang@hust.edu.cn
|
|
|
|
| 1 |
---
|
|
|
|
| 2 |
tags:
|
| 3 |
+
- autonomous-driving
|
| 4 |
+
- video-generation
|
| 5 |
+
- diffusion
|
| 6 |
+
- World-Action-Models
|
|
|
|
| 7 |
---
|
| 8 |
|
| 9 |
<div align="center">
|
| 10 |
|
| 11 |
+
ReWorld: Representation Learning for World Action Models
|
| 12 |
|
| 13 |
+
**The first representation learning framework for autonomous-driving World Action Models**
|
|
|
|
| 14 |
|
| 15 |
[](https://arxiv.org/abs/2606.27504)
|
| 16 |
[](https://xiaomi-research.github.io/ReWorld/)
|
|
|
|
| 30 |
|
| 31 |
---
|
| 32 |
|
| 33 |
+
## Highlights
|
| 34 |
|
| 35 |
- **Better futures** β FVD **81.3 β 61.9** (β23.9%) on nuScenes video generation, with self-guided sampling enabled for free by intermediate supervision.
|
| 36 |
- **Safer plans** β closed-loop PDMS **89.1 β 90.4** on NAVSIM *Navtest*, with best-in-class NC / DAC / TTC among world-model planners β no RL, no test-time scoring.
|
| 37 |
- **Stronger representations** β frozen linear probe on UCF-101 action recognition **68.3% β 80.2%** (+11.9 pts over the DriveLaW baseline).
|
| 38 |
- **Nearly free** β no external encoders, no teacher models, only **+0.3%** per-step Video DiT training cost, and roughly **2Γ faster** convergence from scratch.
|
| 39 |
|
| 40 |
+
## Why ReWorld?
|
| 41 |
|
| 42 |
World Action Models (WAMs) couple future environment prediction with action generation, yet standard training supervises only the output ends of the generation and planning modules. The intermediate representations that carry world knowledge are shaped only indirectly, as byproducts of fitting these outputs. We call this the **representation bottleneck of WAMs**: the world-to-action pathway is not explicitly optimized to be future-predictive, cross-modally grounded, or sensitive to closed-loop behavior quality.
|
| 43 |
|
|
|
|
| 55 |
<img src="assets/reworld_self_guidance.png" width="100%" alt="Self-guided sampling and faster convergence">
|
| 56 |
</div>
|
| 57 |
|
| 58 |
+
## Results at a Glance
|
| 59 |
|
| 60 |
<details open><summary><b>Video generation on nuScenes</b> (val)</summary>
|
| 61 |
|
|
|
|
| 124 |
<br><i>NAVSIM Navtest rollouts (red: ReWorld prediction, green: expert) β straight, left turn, right turn, intersection.</i>
|
| 125 |
</div>
|
| 126 |
|
| 127 |
+
## Checkpoints
|
| 128 |
|
| 129 |
| Folder | File | Role |
|
| 130 |
|:---|:---|:---|
|
|
|
|
| 145 |
|
| 146 |
Please download the **whole repo** (not a single `.safetensors` file). Hugging Face only increments download stats when `config.json` is requested, which `snapshot_download` / `hf download` always fetch.
|
| 147 |
|
| 148 |
+
## Getting Started
|
| 149 |
|
| 150 |
### Installation
|
| 151 |
|
|
|
|
| 212 |
|
| 213 |
> Optional base imitation before Stage 2: `scripts/training/run_videodrive_train.sh` with [`video_model_train_base.yaml`](DriveLaW-Act/navsim/agents/videodrive/configs/ltx_model/video_model_train_base.yaml), or start from the π€ DriveLaW checkpoint.
|
| 214 |
|
| 215 |
+
## Acknowledgments
|
| 216 |
|
| 217 |
+
ReWorld builds on [DriveLaW](https://arxiv.org/abs/2512.23421), [NAVSIM](https://github.com/autonomousvision/navsim), [LTX-Video](https://github.com/Lightricks/LTX-Video), [Diffusers](https://github.com/huggingface/diffusers), and the hard-negative protocol of [BeyondDrive](https://github.com/wjl2244/BeyondDrive). Stage 1 uses intermediate supervision inspired by [Internal Guidance](https://github.com/CVL-UESTC/Internal-Guidance). Thanks to all of them for open-sourcing.
|
| 218 |
|
| 219 |
+
## Citation
|
| 220 |
|
| 221 |
```bibtex
|
| 222 |
@article{xia2026reworld,
|
|
|
|
| 229 |
|
| 230 |
## :mailbox: Contact
|
| 231 |
|
| 232 |
+
Tianze Xia β xiatianze@hust.edu.cn Β· Xinggang Wang β xgwang@hust.edu.cn
|