| --- |
| base_model: |
| - Qwen/Qwen3-VL-4B-Instruct |
| language: |
| - en |
| license: apache-2.0 |
| pipeline_tag: image-to-image |
| library_name: transformers |
| tags: |
| - autonomous-driving |
| - vision-language-action |
| - chain-of-thought |
| - trajectory-prediction |
| - VLA |
| --- |
| |
| # OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation |
|
|
| **[📄 Paper (arXiv)](https://arxiv.org/abs/2604.18486)** | **[💻 GitHub](https://github.com/xiaomi-research/onevl)** | **[🌐 Project Page](https://Xiaomi-Embodied-Intelligence.github.io/OneVL/)** |
|
|
| OneVL is a Vision-Language-Action (VLA) framework for autonomous driving that achieves state-of-the-art trajectory prediction accuracy while matching the inference latency of answer-only autoregressive models. |
|
|
| ## Overview |
|
|
| OneVL addresses the limitations of prior latent Chain-of-Thought (CoT) methods by introducing **dual-modal auxiliary decoders**. These decoders force compact latent tokens to encode both human-readable reasoning and future scene dynamics. During inference, these decoders are discarded, and the latent tokens are prefilled into the context in a single parallel pass, achieving high performance at answer-only speeds. |
|
|
| ### Key Architecture Components |
|
|
| - **Latent Token Interface**: 4 visual and 2 language latent tokens inserted before the response. |
| - **Visual Auxiliary Decoder**: Acts as a world model, predicting future-frame visual tokens (at t+0.5s and t+1.0s). |
| - **Language Auxiliary Decoder**: Reconstructs explicit CoT reasoning text from language latent hidden states. |
| - **Prefill Inference**: Enables 1.5× to 2.3× speedup over explicit autoregressive CoT. |
|
|
| ## Usage |
|
|
| ### Requirements |
|
|
| - Python 3.10+, CUDA GPU (≥16 GB VRAM recommended) |
| - `transformers >= 4.57.0` (required for `Qwen3VLForConditionalGeneration`) |
|
|
| ```bash |
| # Environment Setup |
| uv venv venv/onevl --python 3.12 |
| source venv/onevl/bin/activate |
| pip install -r requirements.txt |
| ``` |
|
|
| ### Inference (Trajectory Prediction Only) |
|
|
| ```bash |
| python infer_onevl.py \ |
| --model_path /path/to/OneVL-checkpoint \ |
| --test_set_path test_data/navsim_test.json \ |
| --image_base_path "" \ |
| --output_path output/navsim/results.json \ |
| --device cuda:0 \ |
| --num_latent 2 --num_latent_vis 4 \ |
| --max_new_tokens 1024 --answer_prefix "[" --prefix_k 0 |
| ``` |
|
|
| For full inference options, including language and visual explanations, please refer to the [GitHub repository](https://github.com/xiaomi-research/onevl). |
|
|
| ## Results |
|
|
| OneVL is the first latent CoT method to surpass explicit autoregressive CoT across all major autonomous driving benchmarks. |
|
|
| | Benchmark | Metric | AR CoT+Answer | OneVL | |
| |---|:---:|:---:|:---:| |
| | **NAVSIM** | PDM-score ↑ | 88.29 | **88.84** | |
| | **ROADWork** | ADE (px) ↓ | 13.18 | **12.49** | |
| | **Impromptu** | ADE (m) ↓ | 1.42 | **1.34** | |
| | **APR1** | ADE (m) ↓ | 2.99 | **2.62** | |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{lu2026onevl, |
| title={OneVL: One-Step Latent Reasoning and Planning with Vision-Language Explanation}, |
| author={Lu, Jinghui and Guan, Jiayi and Huang, Zhijian and Li, Jinlong and Li, Guang and Kong, Lingdong and Li, Yingyan and Wang, Han and Xu, Shaoqing and Luo, Yuechen and others}, |
| journal={arXiv preprint arXiv:2604.18486}, |
| year={2026}, |
| url={https://arxiv.org/abs/2604.18486} |
| } |
| ``` |
|
|
| ## License |
|
|
| This project is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). |
| Model weights are built on [Qwen3-VL-4B-Instruct](https://huggingface.co/Qwen/Qwen3-VL-4B-Instruct) and the visual tokenizer is from [Emu3.5-VisionTokenizer](https://huggingface.co/BAAI/Emu3.5-VisionTokenizer). |
