SimVTA / README.md

Update README.md

d1ade82 verified 14 days ago

4.58 kB

---
license: apache-2.0
base_model: Kevin-thu/Epona
tags:
  - robotics
  - autonomous-driving
  - computer-vision
  - world-model
  - trajectory-planning
---
# SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

[![Paper](https://img.shields.io/badge/-Paper-B31B1B?logo=arxiv&logoColor=white&labelColor=555)](https://arxiv.org/abs/2610.13781)
[![Project Page](https://img.shields.io/badge/Project_Page-8A2BE2?logo=googlechrome&logoColor=white&labelColor=555)](https://opendrivelab.com/SimVTA/)
[![GitHub](https://img.shields.io/badge/-GitHub-181717?logo=github&logoColor=white)](https://github.com/OpenDriveLab/SimVTA)

> [Changhui Jing](https://scholar.google.com/citations?user=B4Nu6pAAAAAJ), [Haochen Tian](https://scholar.google.com/citations?user=DoyuNwEAAAAJ), [Hongchen Li](https://scholar.google.com/citations?user=QO6p6OgAAAAJ), [Tianyu Li](https://sephyli.github.io/), [Caojun Wang](https://scholar.google.com/citations?user=35xHlDUAAAAJ),
> [Zihao Zhang](https://scholar.google.com/citations?user=R2Brvm4AAAAJ), [Yanhao Liu](https://scholar.google.com/citations?user=dYicC-4AAAAJ), [Chengen Xie](https://scholar.google.com/citations?user=q0W3n-cAAAAJ), [Bailin Li](https://www.linkedin.com/in/libailin/), [Hao Zhao](https://sites.google.com/view/fromandto), [Zhenjie Yang](https://scholar.google.com/citations?user=jVlRiUEAAAAJ)
---

![From logged futures to candidate-matched supervision](https://ik.imagekit.io/szpk/SimVTA/overview.png)


## Highlights

- 🚗 **Simulation-grounded alignment:** Render matched future videos for alternative candidate trajectories in reconstructed real-world scenes.
- 🔍 **Selective video supervision:** Prioritize higher-error futures among high-reward candidates and jointly train trajectory planning and video prediction.
- 📈 **Improved planning:** Achieve 91.3 PDMS on NAVSIM v1, 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM, with direct trajectory prediction at inference.

## Overview

**SimVTA** is a simulation-grounded post-training framework for driving world action models. It extends supervision beyond logged trajectories using a 3D Gaussian Splatting simulator, combining trajectory GRPO with a flow-matching video loss that updates shared representations.

Reported results in the paper:

| Benchmark | Metric | SimVTA |
| --- | --- | --- |
| NAVSIM v1 | PDMS | **91.3** (+5.1 over the 86.2 initialization) |
| NAVSIM v2 | EPDMS | **88.2** |
| HUGSIM | Average HD-Score | **38.8** |

## Matched Futures for Alternative Actions

![Simulator-rendered futures for straight driving and lane changes](https://ik.imagekit.io/szpk/SimVTA/matched-futures.png)

**Simulator-rendered futures for going straight (A), changing lanes left (B), and changing lanes right (C).**

## Video-Trajectory Alignment

![Video-trajectory alignment before and after post-training](https://ik.imagekit.io/szpk/SimVTA/video-trajectory-alignment.png?updatedAt=1792686923154)

**Video-trajectory alignment: World Engine renderings compared with SimVTA and Epona predictions.**

## Checkpoint Download

This repository provides `simvta.pkl`, the SimVTA checkpoint post-trained from a NAVSIM-adapted [Epona](https://huggingface.co/Kevin-thu/Epona) initialization.

```bash
pip install -U huggingface_hub
hf download OpenDriveLab-org/SimVTA simvta.pkl --local-dir ./ckpts
```

The checkpoint will be saved to `./ckpts/simvta.pkl`. Code and detailed inference instructions are being prepared for release in the [GitHub repository](https://github.com/OpenDriveLab/SimVTA). Additional pretrained components and dataset preparation will be documented there.

## Acknowledgements

We thank the authors and open-source contributors of the following projects for providing the models, simulation tools, and evaluation benchmarks that support this work:

- [Epona](https://github.com/Kevin-thu/Epona) | [World Engine](https://github.com/OpenDriveLab/WorldEngine) | [NAVSIM](https://github.com/autonomousvision/navsim) | [HUGSIM](https://github.com/hyzhou404/HUGSIM)

We also thank the broader open-source community for its contributions to driving world models, autonomous driving, and 3D Gaussian Splatting.

## License and Citation

The SimVTA release is licensed under [Apache-2.0](LICENSE), subject to the retained upstream MIT notice for Epona in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

If our paper or checkpoint helps your research, please consider citing [our work](https://arxiv.org/abs/2610.13781) and giving the [GitHub repository](https://github.com/OpenDriveLab/SimVTA) a star.

	---
	license: apache-2.0
	base_model: Kevin-thu/Epona
	tags:
	- robotics
	- autonomous-driving
	- computer-vision
	- world-model
	- trajectory-planning
	---

	# SimVTA: Simulation-Grounded Video-Trajectory Alignment for Driving World Action Models

	[![Paper](https://img.shields.io/badge/-Paper-B31B1B?logo=arxiv&logoColor=white&labelColor=555)](https://arxiv.org/abs/2610.13781)
	[![Project Page](https://img.shields.io/badge/Project_Page-8A2BE2?logo=googlechrome&logoColor=white&labelColor=555)](https://opendrivelab.com/SimVTA/)
	[![GitHub](https://img.shields.io/badge/-GitHub-181717?logo=github&logoColor=white)](https://github.com/OpenDriveLab/SimVTA)

	> [Changhui Jing](https://scholar.google.com/citations?user=B4Nu6pAAAAAJ), [Haochen Tian](https://scholar.google.com/citations?user=DoyuNwEAAAAJ), [Hongchen Li](https://scholar.google.com/citations?user=QO6p6OgAAAAJ), [Tianyu Li](https://sephyli.github.io/), [Caojun Wang](https://scholar.google.com/citations?user=35xHlDUAAAAJ),
	> [Zihao Zhang](https://scholar.google.com/citations?user=R2Brvm4AAAAJ), [Yanhao Liu](https://scholar.google.com/citations?user=dYicC-4AAAAJ), [Chengen Xie](https://scholar.google.com/citations?user=q0W3n-cAAAAJ), [Bailin Li](https://www.linkedin.com/in/libailin/), [Hao Zhao](https://sites.google.com/view/fromandto), [Zhenjie Yang](https://scholar.google.com/citations?user=jVlRiUEAAAAJ)
	---

	![From logged futures to candidate-matched supervision](https://ik.imagekit.io/szpk/SimVTA/overview.png)


	## Highlights

	- 🚗 Simulation-grounded alignment: Render matched future videos for alternative candidate trajectories in reconstructed real-world scenes.
	- 🔍 Selective video supervision: Prioritize higher-error futures among high-reward candidates and jointly train trajectory planning and video prediction.
	- 📈 Improved planning: Achieve 91.3 PDMS on NAVSIM v1, 88.2 EPDMS on NAVSIM v2, and 38.8 average HD-Score on HUGSIM, with direct trajectory prediction at inference.

	## Overview

	SimVTA is a simulation-grounded post-training framework for driving world action models. It extends supervision beyond logged trajectories using a 3D Gaussian Splatting simulator, combining trajectory GRPO with a flow-matching video loss that updates shared representations.

	Reported results in the paper:

	\| Benchmark \| Metric \| SimVTA \|
	\| --- \| --- \| --- \|
	\| NAVSIM v1 \| PDMS \| 91.3 (+5.1 over the 86.2 initialization) \|
	\| NAVSIM v2 \| EPDMS \| 88.2 \|
	\| HUGSIM \| Average HD-Score \| 38.8 \|

	## Matched Futures for Alternative Actions

	![Simulator-rendered futures for straight driving and lane changes](https://ik.imagekit.io/szpk/SimVTA/matched-futures.png)

	Simulator-rendered futures for going straight (A), changing lanes left (B), and changing lanes right (C).

	## Video-Trajectory Alignment

	![Video-trajectory alignment before and after post-training](https://ik.imagekit.io/szpk/SimVTA/video-trajectory-alignment.png?updatedAt=1792686923154)

	Video-trajectory alignment: World Engine renderings compared with SimVTA and Epona predictions.

	## Checkpoint Download

	This repository provides `simvta.pkl`, the SimVTA checkpoint post-trained from a NAVSIM-adapted [Epona](https://huggingface.co/Kevin-thu/Epona) initialization.

	```bash
	pip install -U huggingface_hub
	hf download OpenDriveLab-org/SimVTA simvta.pkl --local-dir ./ckpts
	```

	The checkpoint will be saved to `./ckpts/simvta.pkl`. Code and detailed inference instructions are being prepared for release in the [GitHub repository](https://github.com/OpenDriveLab/SimVTA). Additional pretrained components and dataset preparation will be documented there.

	## Acknowledgements

	We thank the authors and open-source contributors of the following projects for providing the models, simulation tools, and evaluation benchmarks that support this work:

	- [Epona](https://github.com/Kevin-thu/Epona) \| [World Engine](https://github.com/OpenDriveLab/WorldEngine) \| [NAVSIM](https://github.com/autonomousvision/navsim) \| [HUGSIM](https://github.com/hyzhou404/HUGSIM)

	We also thank the broader open-source community for its contributions to driving world models, autonomous driving, and 3D Gaussian Splatting.

	## License and Citation

	The SimVTA release is licensed under [Apache-2.0](LICENSE), subject to the retained upstream MIT notice for Epona in [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

	If our paper or checkpoint helps your research, please consider citing [our work](https://arxiv.org/abs/2610.13781) and giving the [GitHub repository](https://github.com/OpenDriveLab/SimVTA) a star.