|
Download README.md from engineerJPark/DriveIK: direct link, hf CLI and curl.
- Browser
- Download file 5.84 kB
-
https://huggingface.co/engineerJPark/DriveIK/resolve/main/README.md
- Command line
-
hf download hf://engineerJPark/DriveIK/README.md
-
curl -L -o README.md https://huggingface.co/engineerJPark/DriveIK/resolve/main/README.md
5.84 kB
| license: apache-2.0 | |
| language: | |
| - en | |
| base_model: Qwen/Qwen2.5-0.5B-Instruct | |
| tags: | |
| - autonomous-driving | |
| - end-to-end-driving | |
| - motion-planning | |
| - vision-language-action | |
| - diffusion | |
| - navsim | |
| <div align="center"> | |
| # DriveIK: Grounding Driving VLA via Inverse Kinematics | |
| **NeurIPS 2026** | |
| Junsung Park, Hyunjung Shim — KAIST, Kim Jaechul Graduate School of AI | |
| <a href="https://engineerjpark.github.io/NeurIPS-2026-DriveIK/"><img src="https://img.shields.io/badge/Project-Page-1f72b8?logo=googlechrome&logoColor=white" alt="Project page"></a> | |
| <a href="https://arxiv.org/abs/2605.21061"><img src="https://img.shields.io/badge/arXiv-2605.21061-b31b1b?logo=arxiv&logoColor=white" alt="Paper (arXiv)"></a> | |
| <a href="https://github.com/engineerJPark/DriveIK"><img src="https://img.shields.io/badge/GitHub-Code-181717?logo=github&logoColor=white" alt="Code (GitHub)"></a> | |
| </div> | |
| Official NAVSIM checkpoints of **DriveIK**, a camera-only driving vision-language-action | |
| (VLA) model built on Qwen2.5-0.5B-Instruct. | |
| Driving VLA models often plan blindly: on ego-predictable examples the planner can rely | |
| on ego status and the driving command instead of the scene. DriveIK anchors trajectory | |
| decoding on visual states. The LLM predicts the future visual state of the scene, and a | |
| separate Inverse Kinematics (IK) network decodes the trajectory from *only* the current | |
| and predicted visual states, so it cannot fall back on ego status or text. | |
| ## Checkpoints | |
| | Folder | Stage | Initialized from | Epochs | NAVSIM-v1 PDMS ↑ | NAVSIM-v2 EPDMS ↑ | | |
| | --- | --- | --- | --- | --- | --- | | |
| | [`DriveIK_NAVSIM_sft`](https://huggingface.co/engineerJPark/DriveIK/tree/main/DriveIK_NAVSIM_sft) | World model + IK (SFT) | Qwen2.5-0.5B-Instruct | 30 | 90.3 | 89.3 | | |
| | [`DriveIK_NAVSIM_rft`](https://huggingface.co/engineerJPark/DriveIK/tree/main/DriveIK_NAVSIM_rft) | GRPO with inline PDM rewards (RFT) | `DriveIK_NAVSIM_sft` | 10 | **92.2** | **90.6** | | |
| Both models are trained on NAVSIM navtrain and evaluated on navtest. Scores are from | |
| the paper; see it for the full tables and ablations. | |
| ## Model | |
| - **LLM:** Qwen2.5-0.5B-Instruct (24 layers, hidden size 896) in a LLaVA-style layout. | |
| - **Visual input:** the three front cameras (`CAM_L0`, `CAM_F0`, `CAM_R0`). Each is | |
| resized to 288×160 and encoded by a frozen | |
| [DINOv3 ViT-S/16](https://huggingface.co/facebook/dinov3-vits16-pretrain-lvd1689m), | |
| which gives 180 patch tokens per camera and 540 × 384 tokens per frame. A linear | |
| projector maps them to the LLM width. No LiDAR or BEV encoder is used. | |
| - **Prompt:** ego status, the 2 s trajectory history, the mission goal and the previous | |
| action, as text. | |
| - **Next visual state prediction:** the LLM regresses the 540 visual tokens of the frame | |
| 4 s ahead. | |
| - **IK network:** a cross-attention conditional diffusion head with 50 sampling steps. | |
| It decodes 8 waypoints (4 s at 0.5 s spacing) from the current and predicted visual | |
| states. | |
| ## Files | |
| Each folder contains: | |
| | File | Contents | | |
| | --- | --- | | |
| | `model.safetensors` | All trained weights (~2 GB, fp32): LLM, visual and ego projectors, IK head | | |
| | `config.json`, `generation_config.json` | Model configuration (`model_type: llava_qwen`) | | |
| | `ik_mode.json` | IK head configuration | | |
| | `vocab.json`, `merges.txt`, `tokenizer_config.json`, `special_tokens_map.json`, `added_tokens.json` | Tokenizer with the DriveIK special tokens | | |
| The DINOv3 encoder is **not** included. It is a gated model: accept its license on the | |
| model page and log in with `huggingface-cli login`. The code downloads it on first use. | |
| These checkpoints use a custom model class and IK head. Load them with the DriveIK code, | |
| not with `transformers.AutoModel`. | |
| ## Usage | |
| Get the code from [github.com/engineerJPark/DriveIK](https://github.com/engineerJPark/DriveIK) | |
| and download the checkpoints into its `checkpoints/` folder: | |
| ```bash | |
| git clone https://github.com/engineerJPark/DriveIK.git && cd DriveIK | |
| huggingface-cli download engineerJPark/DriveIK --local-dir checkpoints | |
| # or a single stage: | |
| huggingface-cli download engineerJPark/DriveIK --include 'DriveIK_NAVSIM_rft/*' --local-dir checkpoints | |
| ``` | |
| Evaluate a checkpoint with the launcher of the same stage. The two recipes differ in | |
| ego-feature width and precision, and a mismatched launcher gives much worse | |
| trajectories. | |
| ```bash | |
| # SFT | |
| DATA_ROOT=/data/driveik \ | |
| SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \ | |
| EVAL_CKPT=checkpoints/DriveIK_NAVSIM_sft \ | |
| bash train_navsim/DriveIK_NAVSIM_sft.sh | |
| # RFT | |
| DATA_ROOT=/data/driveik \ | |
| SENSOR_ROOT=/data/navsim/sensor_blobs/trainval VAL_SENSOR_ROOT=/data/navsim/sensor_blobs/test \ | |
| EVAL_CKPT=checkpoints/DriveIK_NAVSIM_rft \ | |
| bash train_navsim/DriveIK_NAVSIM_rft.sh | |
| ``` | |
| To run the RFT stage from the released SFT model, pass | |
| `SFT_CKPT=checkpoints/DriveIK_NAVSIM_sft` to `train_navsim/DriveIK_NAVSIM_rft.sh`. See the | |
| [code repository](https://github.com/engineerJPark/DriveIK) for installation, data | |
| layout and training. | |
| ## Inputs and limitations | |
| - The models are trained only on NAVSIM (OpenScene / nuPlan) driving logs with three | |
| front cameras. Other camera setups, sensor layouts or regions are untested. | |
| - These are research checkpoints for benchmark evaluation. They are not meant to | |
| control a real vehicle. | |
| ## License | |
| The DriveIK weights are released under the Apache License 2.0. The base model | |
| Qwen2.5-0.5B-Instruct is also Apache 2.0. DINOv3 is distributed separately under its own | |
| license, and the NAVSIM / nuPlan data are subject to their own terms. | |
| ## Citation | |
| ```bibtex | |
| @inproceedings{park2026driveik, | |
| title = {Grounding Driving VLA via Inverse Kinematics}, | |
| author = {Park, Junsung and Shim, Hyunjung}, | |
| booktitle = {Advances in Neural Information Processing Systems (NeurIPS)}, | |
| year = {2026} | |
| } | |
| ``` | |