PersonaDrive

Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation

Mahmoud Srewa · Praneetsai Vasu Iddamsetty · Mohammad Abdullah Al Faruque · Salma Elmalaki

University of California, Irvine · WACV 2027

Project page · Paper (arXiv 2606.12616) · Code · Demo video

A driving agent that drives carefully or aggressively, based on how real people drove.

This is the released driving model of PersonaDrive. PersonaDrive conditions a vision-language-action (VLA) driving agent on demonstrations retrieved from people who were asked to drive in a given style: aggressive, neutral or conservative. One set of weights serves every style. Switching style only changes which database of human demonstrations the retrieval head searches.

Videos, the method and the full results are on the project page.

This checkpoint

This checkpoint is Stage 3 of the PersonaDrive pipeline (in-context fine-tuning). It is SimLingo's driving model, fine-tuned to read K = 2 retrieved context points as few-shot demonstrations before it predicts its own waypoints. Each context point holds an image, control history, command, target points, commentary and executed waypoints. The model gets no style label and infers the style from the demonstrations.

Initialised from	RenzKa/simlingo (epoch=013)
Backbone	OpenGVLab/InternVL2-1B
Parameters	1.36 B in total; about 24 M trained in this stage
Retrieved context points	K = 2
Simulator / benchmark	CARLA 0.9.15 / Bench2Drive

The retrieval head (Stages 1–2) is a separate model of about 2 M trained parameters. It is in the code repository: persona_drive_rag/checkpoints/best_model.pt.

Files

Path	Size	Purpose
personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt	1.96 GB	The model. A single-file PyTorch state dict; load it for evaluation.
personadrive_icl_sft/.hydra/config.yaml	2 KB	The training config. The agent builds the model from it and finds .hydra/ by walking up from the checkpoint path.

pytorch_model.pt was consolidated from the run's DeepSpeed ZeRO-2 checkpoint (last.ckpt). It contains only tensors, so it also loads with torch.load(..., weights_only=True).

The checkpoint is the complete model, not just an adapter. You don't need SimLingo's weights to use it. The weights trained in this stage are stored in fp32 and the frozen ones in bf16:

Part	Tensors	In this stage
LoRA adapters on the language model (r 32, α 64), stored unmerged	336	trained, continuing from SimLingo's adapters
Vision-to-language projector (mlp1)	6	trained
Driving adaptor	10	trained
Waypoint encoder	6	trained
Language-model base weights and language adaptor	294	frozen (as in SimLingo)
Vision encoder (InternViT)	340	frozen (as in SimLingo)

The file is loaded by the PersonaDrive code. It is not a transformers or PEFT checkpoint.

How to use

Requirements:

• Linux x86-64 and CARLA 0.9.15.
• An NVIDIA Ampere, Ada or Hopper GPU, for bf16 and flash-attention 2. Blackwell GPUs need a newer PyTorch than the pinned 2.2.2.
• Installation steps are in the code repository.

In a clone of the repository, with the personadrive environment active:

# the model and its config (~2 GB)
hf download PervasiveAutonomyLab/PersonaDrive --local-dir pretrained \
    --include "personadrive_icl_sft/.hydra/*" "personadrive_icl_sft/checkpoints/epoch=000.ckpt/*"

# one Bench2Drive route on one GPU, aggressive style, K = 2
export AGENT_CKPT=$PWD/pretrained/personadrive_icl_sft/checkpoints/epoch=000.ckpt/pytorch_model.pt
PERSONADRIVE_ICL_PROFILE=fast \
ROUTE_XML=$PERSONADRIVE_ROOT/leaderboard/data/bench2drive_split/bench2drive_00.xml \
OUT_DIR=/tmp/personadrive_route00 \
bash scripts/eval/test_one_lb1_route.sh

• Style: set PERSONADRIVE_ICL_PROFILE to fast (aggressive), default (neutral, normal driving), slow (conservative) or normal (no style; the index built from SimLingo's PDM-lite data). PERSONADRIVE_NUM_ICL=0 turns retrieval off.

• Retrieval data: retrieval needs three things:

  • the retrieval head, included in the code repository;
  • the FAISS indices under persona_drive_rag/, built as in Build the retrieval indices;
  • the camera frames those indices point to.

  The repository's datasets/README.md lists which archives to get for each style. Retrieved frames whose image is missing are skipped. If none can be loaded, the model runs without retrieval.

Results

These are the numbers published in the paper (Bench2Drive, 220 routes, closed loop).

Without style	DS ↑	SR (%) ↑
SimLingo	85.07 ± 0.95	67.27 ± 2.11
HiP-AD	86.77	69.09
PersonaDrive (normal index)	88.95	72.29

PersonaDrive with style (one checkpoint)	DS ↑	Speed (m/s)	Accel. (m/s²)
Conservative	89.62	5.05	3.17
Neutral	87.55	5.52	3.51
Aggressive	88.69	6.05	3.96

From conservative to aggressive, speed rises by 19.8 % and acceleration by 24.9 %. The project page compares these with SimLingo prompted with the style name, StyleDrive and DMW.

Training

• Data:

  • 2,848 PDM-lite routes;
  • 350 routes recorded in the aggressive style (fast/lb1_split);
  • 350 routes recorded in the conservative style (slow/lb1_split).

  Each frame is paired with its K = 2 context points, retrieved from the index of the matching style.

• Settings: LoRA on the language model (r 32, α 64, dropout 0.1) with the vision encoder frozen. AdamW at lr 3e-5 (OneCycle, weight decay 0.1); batch 6 per GPU × 4 GPUs with DeepSpeed ZeRO-2 in bf16; 1 epoch.

• Hardware: 4× NVIDIA L40S (48 GB), about 12 h.

• Reproducing the run: python simlingo_training/train.py experiment=personadrive_icl_sft in the code repository. The composed config of this run is personadrive_icl_sft/.hydra/config.yaml.

Intended use and limitations

• Research use in simulation only (CARLA). The model is not meant to control a real vehicle.
• Where the styles come from: 15 participants who drove under aggressive, neutral and conservative instructions on a driver-in-the-loop rig. Styles are paired at route level, so style is separated from driver identity.
• Speed: with K = 2 the model takes about 0.56 s per simulator tick (paper, offline), so the agent does not run in real time at CARLA's 20 Hz.

License

Apache 2.0, the same license as SimLingo (code and the RenzKa/simlingo weights). The backbone InternVL2-1B is released under the MIT license.

This work was made using the SimLingo dataset, provided by Wayve Technologies Ltd under licence terms available at https://github.com/RenzKa/simlingo. This model was fine-tuned with that dataset, so its terms also restrict the model:

• it may only be distributed free of charge and on an academic or scientific basis;
• it must not be used in, or to assist in, the operation of a vehicle or robot, or for any high-risk purpose.

Citation

@article{srewa2026personadrive,
  title   = {PersonaDrive: Human-Style Retrieval-Augmented VLA Agents for Closed-Loop Driving Simulation},
  author  = {Srewa, Mahmoud and Iddamsetty, Praneetsai Vasu and Al Faruque, Mohammad Abdullah and Elmalaki, Salma},
  journal = {arXiv preprint arXiv:2606.12616},
  year    = {2026}
}

Acknowledgements

This work is supported by the U.S. National Science Foundation under grant 2339266 and partially supported by the UCI ProperAI Institute. PersonaDrive builds directly on SimLingo, CARLA garage, DriveLM / PDM-Lite and Bench2Drive; we thank their authors for releasing their code.