DriveIK / README.md
engineerJPark's picture
Add model card
798fa78 verified
|
Raw History Blame Contribute Delete
5.84 kB
---
license: apache-2.0
language:
- en
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags:
- autonomous-driving
- end-to-end-driving
- motion-planning
- vision-language-action
- diffusion
- navsim
---
<div align="center">
# DriveIK: Grounding Driving VLA via Inverse Kinematics
**NeurIPS 2026**
Junsung Park, Hyunjung Shim — KAIST, Kim Jaechul Graduate School of AI
<a href="https://engineerjpark.github.io/NeurIPS-2026-DriveIK/"><img src="https://img.shields.io/badge/Project-Page-1f72b8?logo=googlechrome&logoColor=white" alt="Project page"></a>
<a href="https://arxiv.org/abs/2605.21061"><img src="https://img.shields.io/badge/arXiv-2605.21061-b31b1b?logo=arxiv&logoColor=white" alt="Paper (arXiv)"></a>
<a href="https://github.com/engineerJPark/DriveIK"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" alt="Code (GitHub)"></a>
</div>
Official NAVSIM checkpoints of **DriveIK**, a camera-only driving vision-language-action
(VLA) model built on Qwen2.5-0.5B-Instruct.
Driving VLA models often plan blindly: on ego-predictable examples the planner can rely
on ego status and the driving command instead of the scene. DriveIK anchors trajectory
decoding on visual states. The LLM predicts the future visual state of the scene, and a
separate Inverse Kinematics (IK) network decodes the trajectory from *only* the current
and predicted visual states, so it cannot fall back on ego status or text.
## Checkpoints
| Folder | Stage | Initialized from | Epochs | NAVSIM-v1 PDMS ↑ | NAVSIM-v2 EPDMS ↑ |
| --- | --- | --- | --- | --- | --- |
| [`DriveIK_NAVSIM_sft`](https://huggingface.co/engineerJPark/DriveIK/tree/main/DriveIK_NAVSIM_sft) | World model + IK (SFT) | Qwen2.5-0.5B-Instruct | 30 | 90.3 | 89.3 |
| [`DriveIK_NAVSIM_rft`](https://huggingface.co/engineerJPark/DriveIK/tree/main/DriveIK_NAVSIM_rft) | GRPO with inline PDM rewards (RFT) | `DriveIK_NAVSIM_sft` | 10 | **92.2** | **90.6** |
Both models are trained on NAVSIM navtrain and evaluated on navtest. Scores are from
the paper; see it for the full tables and ablations.
## Model
- **LLM:** Qwen2.5-0.5B-Instruct (24 layers, hidden size 896) in a LLaVA-style layout.
- **Visual input:** the three front cameras (`CAM_L0`, `CAM_F0`, `CAM_R0`). Each is
resized to 288×160 and encoded by a frozen
[DINOv3 ViT-S/16](https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m),
which gives 180 patch tokens per camera and 540 × 384 tokens per frame. A linear
projector maps them to the LLM width. No LiDAR or BEV encoder is used.
- **Prompt:** ego status, the 2 s trajectory history, the mission goal and the previous
action, as text.
- **Next visual state prediction:** the LLM regresses the 540 visual tokens of the frame
4 s ahead.
- **IK network:** a cross-attention conditional diffusion head with 50 sampling steps.
It decodes 8 waypoints (4 s at 0.5 s spacing) from the current and predicted visual
states.
## Files
Each folder contains:
| File | Contents |
| --- | --- |
| `model.safetensors` | All trained weights (~2 GB, fp32): LLM, visual and ego projectors, IK head |
| `config.json`, `generation_config.json` | Model configuration (`model_type: llava_qwen`) |
| `ik_mode.json` | IK head configuration |
| `vocab.json`, `merges.txt`, `tokenizer_config.json`, `special_tokens_map.json`, `added_tokens.json` | Tokenizer with the DriveIK special tokens |
The DINOv3 encoder is **not** included. It is a gated model: accept its license on the
model page and log in with `huggingface-cli login`. The code downloads it on first use.
These checkpoints use a custom model class and IK head. Load them with the DriveIK code,
not with `transformers.AutoModel`.
## Usage
Get the code from [github.com/engineerJPark/DriveIK](https://github.com/engineerJPark/DriveIK)
and download the checkpoints into its `checkpoints/` folder:
```bash
git clone https://github.com/engineerJPark/DriveIK.git && cd DriveIK
huggingface-cli download engineerJPark/DriveIK --local-dir checkpoints
# or a single stage:
huggingface-cli download engineerJPark/DriveIK --include 'DriveIK_NAVSIM_rft/*' --local-dir checkpoints
```
Evaluate a checkpoint with the launcher of the same stage. The two recipes differ in
ego-feature width and precision, and a mismatched launcher gives much worse
trajectories.
```bash
# SFT
DATA_ROOT=/data/driveik \
SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \
EVAL_CKPT=checkpoints/DriveIK_NAVSIM_sft \
bash train_navsim/DriveIK_NAVSIM_sft.sh
# RFT
DATA_ROOT=/data/driveik \
SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \
EVAL_CKPT=checkpoints/DriveIK_NAVSIM_rft \
bash train_navsim/DriveIK_NAVSIM_rft.sh
```
To run the RFT stage from the released SFT model, pass
`SFT_CKPT=checkpoints/DriveIK_NAVSIM_sft` to `train_navsim/DriveIK_NAVSIM_rft.sh`. See the
[code repository](https://github.com/engineerJPark/DriveIK) for installation, data
layout and training.
## Inputs and limitations
- The models are trained only on NAVSIM (OpenScene / nuPlan) driving logs with three
front cameras. Other camera setups, sensor layouts or regions are untested.
- These are research checkpoints for benchmark evaluation. They are not meant to
control a real vehicle.
## License
The DriveIK weights are released under the Apache License 2.0. The base model
Qwen2.5-0.5B-Instruct is also Apache 2.0. DINOv3 is distributed separately under its own
license, and the NAVSIM / nuPlan data are subject to their own terms.
## Citation
```bibtex
@inproceedings{park2026driveik,
title = {Grounding Driving VLA via Inverse Kinematics},
author = {Park, Junsung and Shim, Hyunjung},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2026}
}
```