xiaomi-research
/

OneVL_NAVSIM

@@ -1,16 +1,17 @@
 ---
-license: apache-2.0
 language:
 - en
 tags:
 - autonomous-driving
 - vision-language-action
 - chain-of-thought
 - trajectory-prediction
 - VLA
-base_model:
-- Qwen/Qwen3-VL-4B-Instruct
-pipeline_tag: image-text-to-text
 ---
 # OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
@@ -37,67 +38,24 @@ OneVL is the **first latent CoT method to surpass explicit autoregressive CoT**
 OneVL augments **Qwen3-VL-4B-Instruct** with three components:
-**Latent Token Interface** — 4 visual latent tokens + 2 language latent tokens are inserted in the assistant response before the answer, using existing vocabulary tokens (no new special tokens added).
-**Visual Auxiliary Decoder** — Predicts future-frame visual tokens at t+0.5s and t+1.0s from visual latent hidden states (using the Emu3.5 IBQ 131k codebook). Acts as a **world model** supervision signal that forces the latent space to encode genuine physical scene dynamics — agent trajectories, road geometry, and environmental change — rather than abstract descriptions.
-**Language Auxiliary Decoder** — Reconstructs explicit CoT reasoning text from language latent hidden states, conditioned on ViT visual features. Recovers 97% of explicit CoT text quality while running at answer-only speed.
-**Prefill Inference** — Both decoders are discarded at inference time. All latent tokens are processed in a single parallel prefill pass; only the trajectory answer is generated autoregressively. This achieves **1.5× speedup over explicit CoT on NAVSIM** and **2.3× on ROADWork**.
-### Three-Stage Training Pipeline
-Training proceeds in three stages to ensure stable joint optimization:
-- **Stage 0**: Main model warmup (trajectory prediction)
-- **Stage 1**: Auxiliary decoder warmup (language + visual decoders independently)
-- **Stage 2**: Joint end-to-end fine-tuning (all components together)
-Staged training is essential — ablation shows that skipping it collapses PDM-score from 88.84 to 67.13.
 ---
 ## Results
-### NAVSIM
 | Method | Model Size | PDM-score ↑ | Latency (s) ↓ | Interpretability |
 |---|:---:|:---:|:---:|:---:|
 | AR Answer | 4B | 87.47 | 4.49 | — |
 | AR CoT+Answer | 4B | 88.29 | 6.58 | Language |
-| COCONUT | 4B | 84.84 | 5.93 | — |
-| CODI | 4B | 83.92 | 8.62 | — |
-| SIM-CoT | 4B | 84.21 | 10.86 | Language |
 | **OneVL** | **4B** | **88.84** | **4.46** | **Vision + Language** |
-### ROADWork
-| Method | ADE (px) ↓ | FDE (px) ↓ | Latency (s) ↓ |
-|---|:---:|:---:|:---:|
-| AR CoT+Answer | 13.18 | 29.98 | 10.74 |
-| **OneVL** | **12.49** | **28.80** | **4.71** |
-### Impromptu
-| Method | ADE (m) ↓ | FDE (m) ↓ | Latency (s) ↓ |
-|---|:---:|:---:|:---:|
-| AR CoT+Answer | 1.42 | 3.96 | 6.84 |
-| **OneVL** | **1.34** | **3.70** | **4.02** |
-### APR1 (Alpamayo-R1)
-| Method | ADE (m) ↓ | FDE (m) ↓ | Latency (s) ↓ |
-|---|:---:|:---:|:---:|
-| AR CoT+Answer | 2.99 | 8.54 | 3.51 |
-| **OneVL** | **2.62** | 7.53 | **3.26** |
-### CoT Text Quality (NAVSIM)
-| Method | Meta Action Acc. ↑ | STS Score ↑ | LLM Judge ↑ | Latency (s) ↓ |
-|---|:---:|:---:|:---:|:---:|
-| AR CoT+Answer | 73.20 | 79.75 | 81.86 | 6.58 |
-| **OneVL** | 71.00 | 78.26 | 79.13 | **4.46** |
-OneVL's language auxiliary decoder recovers 97% of explicit CoT quality at answer-only inference speed.
 ---
@@ -114,7 +72,9 @@ source venv/onevl/bin/activate
 pip install -r requirements.txt
 ```
-### Inference (Trajectory Prediction Only)
 ```bash
 python infer_onevl.py \
@@ -127,7 +87,9 @@ python infer_onevl.py \
     --max_new_tokens 1024 --answer_prefix "[" --prefix_k 0
 ```
-### Inference with Language + Visual Explanation
 ```bash
 python infer_onevl.py \
@@ -144,26 +106,6 @@ python infer_onevl.py \
     --c_thought_visual 4 --max_visual_tokens 2560
 ```
-### Multi-GPU Inference
-```bash
-export MODEL_PATH=/path/to/OneVL-checkpoint
-export TEST_SET_PATH=test_data/navsim_test.json
-export OUTPUT_PATH=output/navsim/navsim_results.json
-bash run_infer.sh
-```
-Per-benchmark scripts are available in `scripts/`:
-```bash
-bash scripts/infer_navsim.sh
-bash scripts/infer_ar1.sh
-bash scripts/infer_roadwork.sh
-bash scripts/infer_impromptu.sh
-```
-For full documentation, evaluation scripts, and data format details, see the [GitHub repository](https://github.com/xiaomi-research/onevl).
 ---
 ## Open-Source Status
@@ -195,4 +137,4 @@ For full documentation, evaluation scripts, and data format details, see the [Gi
 Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
-Model weights are built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) and the visual tokenizer is from [Emu3.5-VisionTokenizer](https://huggingface.co/BAAI/Emu3.5-VisionTokenizer); please refer to their respective licenses as well.

 ---
+base_model:
+- Qwen/Qwen3-VL-4B-Instruct
 language:
 - en
+license: apache-2.0
+pipeline_tag: image-text-to-text
+library_name: transformers
 tags:
 - autonomous-driving
 - vision-language-action
 - chain-of-thought
 - trajectory-prediction
 - VLA
 ---
 # OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation
 OneVL augments **Qwen3-VL-4B-Instruct** with three components:
+- **Latent Token Interface** — 4 visual latent tokens + 2 language latent tokens are inserted in the assistant response before the answer, using existing vocabulary tokens (no new special tokens added).
+- **Visual Auxiliary Decoder** — Predicts future-frame visual tokens at t+0.5s and t+1.0s from visual latent hidden states (using the Emu3.5 IBQ 131k codebook). Acts as a **world model** supervision signal that forces the latent space to encode genuine physical scene dynamics.
+- **Language Auxiliary Decoder** — Reconstructs explicit CoT reasoning text from language latent hidden states, conditioned on ViT visual features.
+- **Prefill Inference** — Both decoders are discarded at inference time. All latent tokens are processed in a single parallel prefill pass; only the trajectory answer is generated autoregressively. This achieves **1.5× speedup over explicit CoT on NAVSIM** and **2.3× on ROADWork**.
 ---
 ## Results
+### NAVSIM Performance
 | Method | Model Size | PDM-score ↑ | Latency (s) ↓ | Interpretability |
 |---|:---:|:---:|:---:|:---:|
 | AR Answer | 4B | 87.47 | 4.49 | — |
 | AR CoT+Answer | 4B | 88.29 | 6.58 | Language |
 | **OneVL** | **4B** | **88.84** | **4.46** | **Vision + Language** |
+OneVL is the first latent CoT method to outperform explicit CoT on driving benchmarks while maintaining answer-only latency.
 ---
 pip install -r requirements.txt
 ```
+### Inference (Trajectory Prediction)
+To run inference for trajectory prediction:
 ```bash
 python infer_onevl.py \
     --max_new_tokens 1024 --answer_prefix "[" --prefix_k 0
 ```
+### Inference with Explanations
+To generate both language and visual explanations:
 ```bash
 python infer_onevl.py \
     --c_thought_visual 4 --max_visual_tokens 2560
 ```
 ---
 ## Open-Source Status
 Released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).
+Model weights are built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) and the visual tokenizer is from [Emu3.5-VisionTokenizer](https://huggingface.co/BAAI/Emu3.5-VisionTokenizer); please refer to their respective licenses as well.