OmniV2X

A generative foundation planner for efficient end-to-end cooperative driving.

OmniV2X treats ego-view images, navigation commands, optional map context, and infrastructure-side V2X messages as independent conditioning sequences for a generative trajectory planner. Instead of fusing dense BEV features, it uses lightweight V2X object tokens and cross-attention injection to keep communication and inference costs low.

Method

OmniV2X uses a frozen visual encoder, lightweight V2X/map encoders, per-source normalization, and a rectified-flow transformer planner. The model is first pretrained on single-agent driving data, then adapted to cooperative DAIR-V2X-Seq data with infrastructure object tokens.

Quick Start

Create the environment and install the package:

conda env create -f environment.yml
conda activate omniv2x
pip install -e .

Set dataset and output paths:

export NAVSIM_DEVKIT_ROOT="$PWD"
export NAVSIM_EXP_ROOT=/path/to/experiments
export OPENSCENE_DATA_ROOT=/path/to/openscene
export NUPLAN_MAPS_ROOT=/path/to/nuplan/maps
export DAIRV2X_DATA_ROOT=/path/to/dair-v2x-seq
export DAIRV2X_MAP_ROOT=/path/to/dair-v2x-seq/maps

The NAVSIM/OpenScene pretraining config expects this layout:

$OPENSCENE_DATA_ROOT/
  navsim_logs/trainval/
  sensor_blobs/trainval/

The helpers under download/ create this layout when run from the target OpenScene data directory.

Download released DAIR-V2X checkpoints:

mkdir -p checkpoints
hf download AndyPJT/OmniV2X \
  omniv2x_dairv2x_no_map.ckpt \
  omniv2x_dairv2x_map.ckpt \
  --local-dir checkpoints

Train the NAVSIM/OpenScene pretraining stage:

bash scripts/train_navsim_pretrain.sh \
  trainer.params.max_epochs=100 \
  dataloader.params.batch_size=32

Fine-tune on DAIR-V2X cooperative data:

export CHECKPOINT_PATH=/path/to/omniv2x_navsim_pretrain.ckpt
bash scripts/finetune_dairv2x.sh \
  trainer.params.max_epochs=500 \
  dataloader.params.batch_size=16

For the map-conditioned variant, use the map agent config:

AGENT_CONFIG=omniv2x_map_flow bash scripts/finetune_dairv2x.sh

Run DAIR-V2X inference:

export CHECKPOINT_PATH=checkpoints/omniv2x_dairv2x_no_map.ckpt
export DAIRV2X_DATA_ROOT=/path/to/dair-v2x-seq
MAP_TOKENS=0 INFRA_BBOX_TOKENS=16 NUM_INFERENCE_STEPS=20 \
  RUN_NAME=omniv2x_no_map_val \
  bash scripts/infer_dairv2x.sh

Run the map-conditioned checkpoint:

export CHECKPOINT_PATH=checkpoints/omniv2x_dairv2x_map.ckpt
export DAIRV2X_DATA_ROOT=/path/to/dair-v2x-seq
export DAIRV2X_MAP_ROOT=/path/to/dair-v2x-seq/maps
MAP_TOKENS=128 INFRA_BBOX_TOKENS=16 NUM_INFERENCE_STEPS=20 \
  RUN_NAME=omniv2x_map_val \
  bash scripts/infer_dairv2x.sh

Inference defaults to --device cuda for paper reproduction. On CPU-only machines, append --device cpu to the scripts/infer_dairv2x.sh command for a small smoke test.

Evaluate DAIR-V2X PDMS from saved trajectory predictions:

export TRAJECTORY_RESULTS=/path/to/trajectory_results.pkl
bash scripts/eval_dairv2x_pdms.sh

The corresponding Hydra configs are:

navsim/planning/script/config/common/agent/omniv2x_base.yaml
navsim/planning/script/config/common/agent/omniv2x_flow.yaml
navsim/planning/script/config/common/agent/omniv2x_map_flow.yaml
navsim/planning/script/config/training/omniv2x_navsim_pretrain.yaml
navsim/planning/script/config/training/omniv2x_dairv2x_finetune.yaml

See docs/CHECKPOINTS.md for the checkpoint/config contract used by the released no-map and map variants.

Training logs to TensorBoard by default under ${NAVSIM_EXP_ROOT}. To use Weights & Biases instead, launch with logger=wandb after logging in to W&B.

Datasets and Weights

This repository does not redistribute NAVSIM/OpenScene, nuPlan maps, or DAIR-V2X-Seq. Download datasets from their official sources and point the environment variables above to your local copies.

Pretrained DAIR-V2X checkpoints are hosted on Hugging Face:

Checkpoint	Link	Intended use
omniv2x_dairv2x_no_map.ckpt	download	SDSM/V2X object-token conditioning without map tokens
omniv2x_dairv2x_map.ckpt	download	SDSM + MAP conditioning with 128 map tokens

Results

Paper-reported DAIR-V2X cooperative planning results:

Model	V2X message	Cost (BPS)	Avg. L2 (m, lower better)	3s Avg. Collision (%, lower better)	PDMS (higher better)
OmniV2X	SDSM	1,408	0.86 +/- 0.01	0.06 +/- 0.02	88.33 +/- 0.23
OmniV2X (w/ Map)	SDSM + MAP	25,792	0.86 +/- 0.00	0.01 +/- 0.01	89.87 +/- 0.18

The paper also reports 35 ms inference latency and 550 MB GPU memory usage for the non-map OmniV2X setting. Single-checkpoint DAIR-V2X validation metrics for the released weights are listed in docs/CHECKPOINTS.md.

Repository Layout

assets/figures/                         Paper figures used by this README
download/                               Public NAVSIM/OpenScene download helpers
navsim/agents/omniv2x/core/             Agent, feature builders, and planner wrapper
navsim/agents/omniv2x/models/           Vision encoder, Q-Former, and flow/AR planner modules
navsim/agents/omniv2x/scripts/          Inference and DAIR-V2X PDMS evaluation code
navsim/agents/omniv2x/utils/            Training loop, Lightning module, and logging helpers
navsim/planning/script/config/          Hydra configs retained for OmniV2X training/evaluation
docs/                                  Checkpoint and configuration notes
scripts/                                Public launchers for pretraining, fine-tuning, inference, and PDMS
tests/                                  Lightweight regression tests

Citation

@article{peng2026omniv2x,
  title={OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving},
  author={Peng, Juntong and Lu, Juanwu and Zhou, Yupeng and Cui, Can and Chen, Yaobin and Wang, Ziran},
  journal={arXiv preprint arXiv:2606.21165},
  year={2026}
}

Acknowledgements

OmniV2X builds on the NAVSIM planning stack and evaluates on DAIR-V2X-Seq-style cooperative driving data. We thank the maintainers of NAVSIM, nuPlan, DAIR-V2X, PyTorch, Lightning, Hydra, and the open-source driving research community.

License

This project is released under the Apache-2.0 license. Dataset licenses and checkpoint terms are governed by their respective providers.