zczhao commited on
Commit
44dfb1e
Β·
verified Β·
1 Parent(s): ac94008

Upload README.md

Browse files
Files changed (1) hide show
  1. README.md +171 -1
README.md CHANGED
@@ -1,3 +1,173 @@
 
1
  ---
2
- license: apache-2.0
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ```markdown
2
  ---
3
+ license: mit
4
+ pipeline_tag: other
5
+ base_model: Wan-AI/Wan2.2-TI2V-5B
6
+ tags:
7
+ - pytorch
8
+ - autonomous-driving
9
+ - end-to-end-autonomous-driving
10
+ - world-action-model
11
+ - trajectory-planning
12
+ - navsim
13
  ---
14
+
15
+ # SimWAM: A Simple World Action Model for End-to-End Autonomous Driving
16
+
17
+ SimWAM is a World-Action Model for end-to-end autonomous driving. It transfers video-dynamics priors to trajectory prediction by jointly training a pretrained video expert and a lightweight action diffusion transformer with flow matching.
18
+
19
+ An isolated attention design prevents the action branch from depending on future frames. The video-generation branch is therefore not required during deployment, allowing the trained action expert to operate as a direct trajectory planner.
20
+
21
+ This repository contains the official released checkpoints for the SimWAM project.
22
+
23
+ ## Task
24
+
25
+ SimWAM performs end-to-end autonomous-driving trajectory planning on NAVSIM.
26
+
27
+ The released model uses a single front-facing camera and predicts a future ego trajectory directly from the current observation and conditioning information. The project supports supervised imitation learning and FlowGRPO reinforcement-learning fine-tuning.
28
+
29
+ Because autonomous-driving trajectory planning does not currently map cleanly to a standard Hugging Face inference pipeline, this repository uses `pipeline_tag: other`.
30
+
31
+ ## Inputs and Outputs
32
+
33
+ ### Inputs
34
+
35
+ The NAVSIM configuration uses:
36
+
37
+ - A front-facing RGB camera observation resized to `384 Γ— 672`
38
+ - Precomputed text or scene-context embeddings
39
+ - An 8-dimensional ego-state representation
40
+ - Optional trajectory-planning configuration supplied by the official codebase
41
+
42
+ ### Outputs
43
+
44
+ The model predicts:
45
+
46
+ - An eight-step future trajectory
47
+ - Three values per trajectory step in the NAVSIM action representation
48
+ - A tensor with shape `[8, 3]` for one evaluated sample
49
+
50
+ The released checkpoints use normalized trajectory targets. Dataset statistics and the official NAVSIM evaluation code should be used to convert predictions into the required evaluation representation.
51
+
52
+ ## Checkpoints
53
+
54
+ | File | Description |
55
+ | --- | --- |
56
+ | `weights/SimWAM.pt` | Supervised SimWAM checkpoint |
57
+ | `weights/SimWAM-RL.pt` | SimWAM checkpoint fine-tuned with FlowGRPO reinforcement learning |
58
+
59
+ These files are custom PyTorch checkpoints and are not directly loadable with `transformers.AutoModel`.
60
+
61
+ ## Download
62
+
63
+ Download the complete repository:
64
+
65
+ ```python
66
+ from huggingface_hub import snapshot_download
67
+
68
+ local_path = snapshot_download(
69
+ repo_id="H-EmbodVis/SimWAM"
70
+ )
71
+
72
+ print(local_path)
73
+ ```
74
+
75
+ Download an individual checkpoint:
76
+
77
+ ```python
78
+ from huggingface_hub import hf_hub_download
79
+
80
+ checkpoint_path = hf_hub_download(
81
+ repo_id="H-EmbodVis/SimWAM",
82
+ filename="weights/SimWAM.pt",
83
+ )
84
+
85
+ print(checkpoint_path)
86
+ ```
87
+
88
+ Command-line download:
89
+
90
+ ```bash
91
+ hf download H-EmbodVis/SimWAM --local-dir ./SimWAM-checkpoints
92
+ ```
93
+
94
+ ## Usage
95
+
96
+ Clone and install the official implementation:
97
+
98
+ ```bash
99
+ git clone https://github.com/H-EmbodVis/SimWAM.git
100
+ cd SimWAM
101
+
102
+ conda create -n simwam python=3.10 -y
103
+ conda activate simwam
104
+
105
+ python -m pip install -r requirements.txt
106
+ python -m pip install -e navsim --no-deps
107
+ python -m pip install -e . --no-deps
108
+ ```
109
+
110
+ Place the downloaded checkpoint under `weights/`, then run the official NAVSIM evaluation script:
111
+
112
+ ```bash
113
+ CKPT=./weights/SimWAM.pt \
114
+ TASK=navsim_uncond_front_384x672_1e-4 \
115
+ NPROC_PER_NODE=1 \
116
+ bash experiments/navsim/run_eval_navsim.sh
117
+ ```
118
+
119
+ NAVSIM, nuPlan maps, sensor blobs, text embeddings, normalization statistics, and other required resources must be prepared according to the official project documentation.
120
+
121
+ ## Repository Contents
122
+
123
+ ```text
124
+ .
125
+ β”œβ”€β”€ config.json
126
+ β”œβ”€β”€ README.md
127
+ └── weights
128
+ β”œβ”€β”€ SimWAM.pt
129
+ └── SimWAM-RL.pt
130
+ ```
131
+
132
+ - `README.md` provides the Hugging Face model card and metadata.
133
+ - `config.json` identifies the repository as a custom PyTorch model repository.
134
+ - `weights/SimWAM.pt` contains the supervised checkpoint.
135
+ - `weights/SimWAM-RL.pt` contains the reinforcement-learning checkpoint.
136
+
137
+ ## Performance
138
+
139
+ The project reports a PDMS score of `91.5` on the NAVSIM `navtest` benchmark using a single front-facing camera.
140
+
141
+ For complete evaluation settings, comparisons, and reproduction instructions, refer to the official GitHub repository.
142
+
143
+ ## Limitations
144
+
145
+ - The checkpoints require the official SimWAM implementation.
146
+ - The model is designed for the NAVSIM data format and evaluation pipeline.
147
+ - Performance outside the training and evaluation distributions is not guaranteed.
148
+ - Predictions must not be directly deployed on a physical vehicle without additional validation, safety systems, and human oversight.
149
+ - NAVSIM, nuPlan, OpenScene, Wan2.2, and other upstream resources retain their own licenses and usage terms.
150
+
151
+ ## GitHub
152
+
153
+ Official implementation:
154
+
155
+ https://github.com/H-EmbodVis/SimWAM
156
+
157
+ ## License
158
+
159
+ The SimWAM source repository is released under the MIT License. Third-party code, datasets, pretrained models, and other assets remain subject to their respective licenses.
160
+
161
+ ## Citation
162
+
163
+ ```bibtex
164
+ @article{zhao2026simwam,
165
+ title = {A Simple World Action Model for End-to-End Autonomous Driving},
166
+ author = {Zhao, Zongchuang and Zhou, Xin and Xu, Tianyang and
167
+ Sun, Zhengyang and Zhou, Kaixuan and Li, Honglin and
168
+ Liang, Dingkang and Bai, Xiang},
169
+ journal = {arXiv preprint},
170
+ year = {2026}
171
+ }
172
+ ```
173
+ ```