| --- |
| license: apache-2.0 |
| base_model: RenzKa/simlingo |
| base_model_relation: finetune |
| pipeline_tag: robotics |
| language: |
| - en |
| tags: |
| - autonomous-driving |
| - carla |
| - bench2drive |
| - vision-language-action |
| - retrieval-augmented |
| - in-context-learning |
| - driving-style |
| - lora |
| --- |
| |
| <div align="center"> |
|
|
| # PersonaDrive |
|
|
| ### Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation |
|
|
| [Mahmoud Srewa](https://mosrewa.github.io/) · Praneetsai Vasu Iddamsetty · Mohammad Abdullah Al Faruque · Salma Elmalaki |
|
|
| University of California, Irvine · WACV 2027 |
|
|
| **[Project page](https://pervasiveautonomylab.github.io/personadrive-landpage/)** · |
| [Paper (arXiv 2606.12616)](https://arxiv.org/abs/2606.12616) · |
| [Code](https://github.com/PervasiveAutonomyLab/PersonaDrive) · |
| [Demo video](https://pervasiveautonomylab.github.io/personadrive-landpage/static/videos/demo_full.mp4) |
|
|
| <a href="https://pervasiveautonomylab.github.io/personadrive-landpage/"><img src="assets/hero_teaser.webp" width="88%" alt="PersonaDrive teaser"></a> |
|
|
| **A driving agent that drives carefully or aggressively, based on how real people drove.** |
|
|
| </div> |
|
|
| This is the released driving model of PersonaDrive. PersonaDrive conditions a vision-language-action (VLA) driving agent on demonstrations retrieved from people who were asked to drive in a given style: aggressive, neutral or conservative. **One set of weights serves every style.** Switching style only changes which database of human demonstrations the retrieval head searches. |
|
|
| Videos, the method and the full results are on the **[project page](https://pervasiveautonomylab.github.io/personadrive-landpage/)**. |
|
|
| <p align="center"><img src="assets/architecture.png" width="95%" alt="PersonaDrive architecture"></p> |
|
|
| ## This checkpoint |
|
|
| This checkpoint is Stage 3 of the PersonaDrive pipeline (in-context fine-tuning). It is SimLingo's driving model, fine-tuned to read **K = 2 retrieved context points** as few-shot demonstrations before it predicts its own waypoints. Each context point holds an image, control history, command, target points, commentary and executed waypoints. The model gets no style label and infers the style from the demonstrations. |
|
|
| | | | |
| |---|---| |
| | Initialised from | [RenzKa/simlingo](https://huggingface.co/RenzKa/simlingo) (`epoch=013`) | |
| | Backbone | [OpenGVLab/InternVL2-1B](https://huggingface.co/OpenGVLab/InternVL2-1B) | |
| | Parameters | 1.36 B in total; about 24 M trained in this stage | |
| | Retrieved context points | K = 2 | |
| | Simulator / benchmark | CARLA 0.9.15 / Bench2Drive | |
|
|
| The retrieval head (Stages 1–2) is a separate model of about 2 M trained parameters. It is in the code repository: [`persona_drive_rag/checkpoints/best_model.pt`](https://github.com/PervasiveAutonomyLab/PersonaDrive/blob/main/persona_drive_rag/checkpoints/best_model.pt). |
|
|
| ## Files |
|
|
| | Path | Size | Purpose | |
| |---|---:|---| |
| | `personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt` | 1.96 GB | **The model.** A single-file PyTorch state dict; load it for evaluation. | |
| | `personadrive_icl_sft/.hydra/config.yaml` | 2 KB | The training config. The agent builds the model from it and finds `.hydra/` by walking up from the checkpoint path. | |
| | `retrieval/normal/driving_index.faiss` | 221 MB | Vectors of the `normal` (no-style) retrieval index: 54,040 frames of the SimLingo dataset, embedded with the released retrieval head. | |
| | `retrieval/normal/frames.json.gz` | 1 MB | The frames those vectors belong to. The SimLingo data itself is not included; `scripts/retrieval/fill_normal_index.py` in the code repository completes the index from your SimLingo download. | |
| | `retrieval/normal/driving_index.info.json` | < 1 KB | Index type and size. | |
|
|
| `pytorch_model.pt` was consolidated from the run's DeepSpeed ZeRO-2 checkpoint (`last.ckpt`). It contains only tensors, so it also loads with `torch.load(..., weights_only=True)`. |
|
|
| The checkpoint is the **complete model**, not just an adapter. You don't need SimLingo's weights to use it. The weights trained in this stage are stored in fp32 and the frozen ones in bf16: |
|
|
| | Part | Tensors | In this stage | |
| |---|---:|---| |
| | LoRA adapters on the language model (r 32, α 64), stored unmerged | 336 | trained, continuing from SimLingo's adapters | |
| | Vision-to-language projector (`mlp1`) | 6 | trained | |
| | Driving adaptor | 10 | trained | |
| | Waypoint encoder | 6 | trained | |
| | Language-model base weights and language adaptor | 294 | frozen (as in SimLingo) | |
| | Vision encoder (InternViT) | 340 | frozen (as in SimLingo) | |
|
|
| The file is loaded by the PersonaDrive code. It is not a `transformers` or PEFT checkpoint. |
|
|
| ## How to use |
|
|
| **Requirements:** |
| - Linux x86-64 and CARLA 0.9.15. |
| - An NVIDIA Ampere, Ada or Hopper GPU, for bf16 and flash-attention 2. Blackwell GPUs need a newer PyTorch than the pinned 2.2.2. |
| - Installation steps are in the [code repository](https://github.com/PervasiveAutonomyLab/PersonaDrive). |
|
|
| In a clone of the repository, with the `personadrive` environment active: |
|
|
| ```bash |
| # the model and its config (~2 GB) |
| hf download PervasiveAutonomyLab/PersonaDrive --local-dir pretrained \ |
| --include "personadrive_icl_sft/.hydra/*" "personadrive_icl_sft/checkpoints/epoch=000.ckpt/*" |
| |
| # one Bench2Drive route on one GPU, aggressive style, K = 2 |
| export AGENT_CKPT=$PWD/pretrained/personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt |
| PERSONADRIVE_ICL_PROFILE=fast \ |
| ROUTE_XML=$PERSONADRIVE_ROOT/leaderboard/data/bench2drive_split/bench2drive_00.xml \ |
| OUT_DIR=/tmp/personadrive_route00 \ |
| bash scripts/eval/test_one_lb1_route.sh |
| ``` |
|
|
| - **Style:** set `PERSONADRIVE_ICL_PROFILE` to `fast` (aggressive), `default` (neutral, normal driving), `slow` (conservative) or `normal` (no style; the index built from SimLingo's PDM-lite data). `PERSONADRIVE_NUM_ICL=0` turns retrieval off. |
| - **Retrieval data:** retrieval needs three things: |
| - the retrieval head, included in the [code repository](https://github.com/PervasiveAutonomyLab/PersonaDrive); |
| - the FAISS indices under `persona_drive_rag/`, built as in [Build the retrieval indices](https://github.com/PervasiveAutonomyLab/PersonaDrive#3-build-the-retrieval-indices) (the `normal` index is completed from `retrieval/normal/` in this repository); |
| - the camera frames those indices point to. |
|
|
| The repository's [`datasets/README.md`](https://github.com/PervasiveAutonomyLab/PersonaDrive/blob/main/datasets/README.md) lists which archives to get for each style. Retrieved frames whose image is missing are skipped. If none can be loaded, the model runs without retrieval. |
|
|
| ## Results |
|
|
| These are the numbers published in the paper (Bench2Drive, 220 routes, closed loop). |
|
|
| | Without style | DS ↑ | SR (%) ↑ | |
| |---|---:|---:| |
| | SimLingo | 85.07 ± 0.95 | 67.27 ± 2.11 | |
| | HiP-AD | 86.77 | 69.09 | |
| | **PersonaDrive** (`normal` index) | **88.95** | **72.29** | |
|
|
| | PersonaDrive with style (one checkpoint) | DS ↑ | Speed (m/s) | Accel. (m/s²) | |
| |---|---:|---:|---:| |
| | Conservative | 89.62 | 5.05 | 3.17 | |
| | Neutral | 87.55 | 5.52 | 3.51 | |
| | Aggressive | 88.69 | 6.05 | 3.96 | |
|
|
| From conservative to aggressive, speed rises by 19.8 % and acceleration by 24.9 %. The project page compares these with SimLingo prompted with the style name, StyleDrive and DMW. |
|
|
| ## Training |
|
|
| - **Data:** |
| - 2,848 PDM-lite routes; |
| - 350 routes recorded in the aggressive style (`fast/lb1_split`); |
| - 350 routes recorded in the conservative style (`slow/lb1_split`). |
|
|
| Each frame is paired with its K = 2 context points, retrieved from the index of the matching style. |
| - **Settings:** LoRA on the language model (r 32, α 64, dropout 0.1) with the vision encoder frozen. AdamW at lr 3e-5 (OneCycle, weight decay 0.1); batch 6 per GPU × 4 GPUs with DeepSpeed ZeRO-2 in bf16; 1 epoch. |
| - **Hardware:** 4× NVIDIA L40S (48 GB), about 12 h. |
| - **Reproducing the run:** `python simlingo_training/train.py experiment=personadrive_icl_sft` in the [code repository](https://github.com/PervasiveAutonomyLab/PersonaDrive#full-pipeline). The composed config of this run is `personadrive_icl_sft/.hydra/config.yaml`. |
|
|
| ## Intended use and limitations |
|
|
| - **Research use in simulation only (CARLA).** The model is not meant to control a real vehicle. |
| - **Where the styles come from:** 15 participants who drove under aggressive, neutral and conservative instructions on a driver-in-the-loop rig. Styles are paired at route level, so style is separated from driver identity. |
| - **Speed:** with K = 2 the model takes about 0.56 s per simulator tick (paper, offline), so the agent does not run in real time at CARLA's 20 Hz. |
|
|
| ## License |
|
|
| Apache 2.0, the same license as SimLingo (code and the [RenzKa/simlingo](https://huggingface.co/RenzKa/simlingo) weights). The backbone [InternVL2-1B](https://huggingface.co/OpenGVLab/InternVL2-1B) is released under the MIT license. |
|
|
| This work was made using the SimLingo dataset, provided by Wayve Technologies Ltd under licence terms available at https://github.com/RenzKa/simlingo. This model was fine-tuned with that dataset, so its terms also restrict the model: |
| - it may only be distributed free of charge and on an academic or scientific basis; |
| - it must not be used in, or to assist in, the operation of a vehicle or robot, or for any high-risk purpose. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{srewa2026personadrive, |
| title = {PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation}, |
| author = {Srewa, Mahmoud and Iddamsetty, Praneetsai Vasu and Al Faruque, Mohammad Abdullah and Elmalaki, Salma}, |
| journal = {arXiv preprint arXiv:2606.12616}, |
| year = {2026} |
| } |
| ``` |
|
|
| ## Acknowledgements |
|
|
| This work is supported by the U.S. National Science Foundation under grant 2339266 and partially supported by the UCI ProperAI Institute. PersonaDrive builds directly on SimLingo, CARLA garage, DriveLM / PDM-Lite and Bench2Drive; we thank their authors for releasing their code. |
|
|
