README.md

---
base_model:
- Wan-AI/Wan2.2-TI2V-5B
license: other
pipeline_tag: robotics
tags:
- pytorch
- end-to-end-autonomous-driving
- world-action-model
- navsim
---

<h1 align="center">SimWAM: A Simple World Action Model for End-to-End Autonomous Driving</h1>

<div align="center">
  <a href="https://github.com/H-EmbodVis/SimWAM"><img src="https://img.shields.io/badge/GitHub-Repository-green?logo=github" alt="GitHub repository"></a>
  <a href="https://huggingface.co/H-EmbodVis/SimWAM"><img src="https://img.shields.io/badge/Hugging%20Face-Model-yellow?logo=huggingface" alt="Hugging Face model"></a>
  <a href="https://huggingface.co/papers/2608.07468"><img src="https://img.shields.io/badge/Paper-arXiv-b31b1b?logo=arxiv&logoColor=white" alt="Paper"></a>


  <h5>
    <em>
      Zongchuang Zhao<sup>1</sup>,
      Xin Zhou<sup>1</sup>,
      Tianyang Xu<sup>1</sup>,
      Zhengyang Sun<sup>1</sup>,
      Kaixuan Zhou<sup>2</sup>,
      Honglin Li<sup>2</sup>,
      Dingkang Liang<sup>1†</sup>,
      Xiang Bai<sup>1</sup>
    </em>
  </h5>

  <sup>1</sup> Huazhong University of Science and Technology<br>
  <sup>2</sup> Dongfeng Research & Development Institute<br>
  <sup>†</sup> Project lead
</div>

## 🔍 Overview

**SimWAM** is a World-Action Model for end-to-end autonomous driving. It transfers video-dynamics priors to trajectory prediction by jointly training a pretrained video expert and a lightweight action diffusion transformer with flow matching.

An isolated attention mask prevents the action branch from accessing future-frame tokens. As a result, the video-generation branch is used only as a training signal and can be removed at inference time, leaving a self-contained action expert that directly predicts future driving trajectories.

This repository contains the official supervised and reinforcement-learning checkpoints for SimWAM on NAVSIM.

<details>
  <summary><b>Method Summary</b></summary>

SimWAM jointly trains a pretrained video expert and an action expert through a unified attention interface. Both experts attend to the current visual observation, while future-video tokens and action tokens remain isolated from each other.

This design transfers motion and traffic-dynamics knowledge from video prediction to trajectory planning without requiring future-scene generation during deployment. The action expert can subsequently be optimized using FlowGRPO and the compositional NAVSIM PDM reward.

</details>

### ✨ Key Ideas

- Joint flow-matching training of a pretrained video expert and a lightweight action diffusion transformer.
- Isolated attention between future-video tokens and action tokens.
- Direct trajectory prediction without future-video generation at inference time.
- Action-only reinforcement-learning fine-tuning with FlowGRPO.
- Independent scaling of the video and action experts without shared model parameters.

## 🎯 Task, Inputs, and Outputs

SimWAM performs end-to-end ego-trajectory planning from a single front-facing camera observation and vehicle-conditioning information.

Because autonomous-driving trajectory planning does not correspond to a standard Hugging Face inference pipeline, this model repository uses `pipeline_tag: robotics`.

| Benchmark | Inputs | Output | Prediction horizon | Sensor configuration |
| --- | --- | --- | ---: | --- |
| NAVSIM | Front RGB image, scene-context embedding, 8-D ego state | Normalized future ego trajectory with shape `[8, 3]` | 8 steps | One front-facing camera at `384 × 672` |

### Inputs

The official NAVSIM configuration uses:

- One front-facing RGB camera observation resized to `384 × 672`
- Precomputed text or scene-context embeddings
- An 8-dimensional ego-state representation
- Model and evaluation configuration supplied by the official SimWAM codebase

### Outputs

The action expert predicts:

- An eight-step future ego trajectory
- Three trajectory values for each predicted step
- A tensor with shape `[8, 3]` for each evaluated sample

The released checkpoints predict normalized trajectory values. The official dataset statistics and NAVSIM evaluation pipeline are required to convert predictions into the benchmark action representation.

## 📦 Released Checkpoints

| Training stage | Checkpoint | Description |
| --- | --- | --- |
| Supervised learning | `weights/SimWAM.pt` | SimWAM checkpoint trained with joint video-action flow matching |
| Reinforcement learning | `weights/SimWAM-RL.pt` | SimWAM checkpoint fine-tuned with FlowGRPO and the NAVSIM PDM reward |

The released files are custom PyTorch checkpoints. They require the official SimWAM implementation and cannot be loaded directly with `transformers.AutoModel.from_pretrained()`.

## 🛠️ Usage

You can download the complete model repository using either the Hugging Face CLI or the Python SDK.

### Option 1: Hugging Face CLI

```bash
pip install -U huggingface_hub

hf download H-EmbodVis/SimWAM \
  --repo-type model \
  --local-dir SimWAM-model
```

### Option 2: Python SDK

```python
from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="H-EmbodVis/SimWAM",
    repo_type="model",
    local_dir="SimWAM-model",
)
```

To download only the supervised checkpoint:

```python
from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="H-EmbodVis/SimWAM",
    repo_type="model",
    filename="weights/SimWAM.pt",
)

print(checkpoint_path)
```

### Installation and Evaluation

Clone and install the official implementation:

```bash
git clone https://github.com/H-EmbodVis/SimWAM.git
cd SimWAM

conda create -n simwam python=3.10 -y
conda activate simwam

python -m pip install -r requirements.txt
python -m pip install -e navsim --no-deps
python -m pip install -e . --no-deps
```

Place the downloaded checkpoints in the `weights/` directory.

A one-sample evaluation of the supervised checkpoint can be launched with:

```bash
CKPT=./weights/SimWAM.pt \
TASK=navsim_uncond_front_384x672_1e-4 \
NPROC_PER_NODE=1 \
bash experiments/navsim/run_eval_navsim.sh \
  EVALUATION.max_samples=1 \
  EVALUATION.num_inference_steps=2 \
  EVALUATION.save_videos=false
```

NAVSIM, nuPlan/OpenScene data, maps, sensor blobs, precomputed embeddings, and normalization statistics must be prepared according to the official GitHub documentation.

## 📈 Performance

Using one front-facing camera at `384 × 672`, SimWAM reports a **91.5 PDMS** score on the NAVSIM `navtest` benchmark after reinforcement-learning fine-tuning.

| Configuration | PDMS |
| --- | ---: |
| Action-only baseline | 86.6 |
| Joint video-action training | 90.3 |
| Joint training with FlowGRPO | **91.5** |

For complete evaluation settings and comparisons, refer to the official implementation:

https://github.com/H-EmbodVis/SimWAM

## 📂 Repository Contents

| Path | Description |
| --- | --- |
| `weights/SimWAM.pt` | Supervised SimWAM checkpoint |
| `weights/SimWAM-RL.pt` | FlowGRPO reinforcement-learning checkpoint |
| `config.json` | Minimal Hugging Face model metadata |
| `README.md` | Hugging Face model card |
| `.gitattributes` | Hugging Face/Xet large-file configuration |

Expected repository structure:

```text
.
├── .gitattributes
├── config.json
├── README.md
└── weights
    ├── SimWAM.pt
    └── SimWAM-RL.pt
```

## ⚠️ Limitations and Safety

- The checkpoints require the official SimWAM implementation and its pinned dependencies.
- The model is designed for the NAVSIM data format and evaluation protocol.
- Performance outside the training and evaluation distributions is not guaranteed.
- The model may produce unsafe or invalid trajectories under unfamiliar road, weather, traffic, or sensor conditions.
- Predictions must not be deployed directly on a physical vehicle without extensive validation, safety constraints, fallback systems, and human oversight.
- NAVSIM, nuPlan, OpenScene, Wan2.2, and other upstream resources retain their respective licenses and distribution terms.

## 🔗 Links

- **Paper:** https://huggingface.co/papers/2608.07468
- **Hugging Face:** https://huggingface.co/H-EmbodVis/SimWAM
- **GitHub:** https://github.com/H-EmbodVis/SimWAM
- **NAVSIM:** https://github.com/autonomousvision/navsim
- **Wan2.2:** https://huggingface.co/Wan-AI/Wan2.2-TI2V-5B

## ⚖️ License

The SimWAM source repository is released under the MIT License.

Third-party code, datasets, pretrained models, and other upstream assets remain subject to their respective licenses and usage terms.

## 📖 Citation

If you find SimWAM useful in your research, please consider citing:

```bibtex
@article{zhao2026simwam,
  title={SimWAM: A Simple World Action Model for End-to-End Autonomous Driving}, 
  author={Zongchuang Zhao and Xin Zhou and Tianyang Xu and Zhengyang Sun and Kaixuan Zhou and Honglin Li and Dingkang Liang and Xiang Bai},
  journal={arXiv preprint arXiv:2608.07468},
  year = {2026}
}
```