license: mit
pipeline_tag: image-to-video
Orbis 2: A Hierarchical World Model for Driving
Sudhanshu Mittal*, Arian Mousakhan*, Silvio Galesso*, Karim Farid, Johannes Dienert, Rajat Sahay, Thomas Brox
University of Freiburg
* Main contributors
Orbis-2 is a hierarchical driving world model that generates long-horizon future video conditioned on past frames and an optional steering signal. A frozen low-frame-rate L2 predictor provides abstract long-range context, while the L1 detail predictor autoregressively generates high-frame-rate future frames. Steering can be given either as raw ego-motion values (speed and yaw rate) or as a 2D trajectory that the model should follow.
Installation
git clone https://github.com/lmb-freiburg/orbis2.git
cd orbis2
conda env create -f environment.yml
conda activate orbis2_env
Video Generation (Roll-out)
evaluate/rollout_demo_v2.py rolls out the world model from a single input video: it samples the L1 (high-rate) and L2 (low-rate, further back in time) context windows directly from the video, then autoregressively generates future frames.
Set the environment variable ORBIS2_MODELS_DIR with the path of the checkpoints folder, e.g.:
export ORBIS2_MODELS_DIR=./orbis2
To roll out the model using an input context video:
python evaluate/rollout_demo_v2.py \
--video /path/to/input_video.mp4 \
--output_dir rollout
To roll out with trajectory steering and an ego-centric trajectory overlay, specify an input trajectory file:
python evaluate/rollout_demo_v2.py \
--video /path/to/input_video.mp4 \
--trajectory_file trajectory.csv \
--vis_mode trajectory_ego \
--output_dir rollout_traj
BibTeX
@article{orbis2_2026,
author = {Mittal, Sudhanshu and Mousakhan, Arian and Galesso, Silvio and
Farid, Karim and Dienert, Johannes and Sahay, Rajat and Brox, Thomas},
title = {Orbis 2: A Hierarchical World Model for Driving},
journal = {arXiv preprint arXiv:2607.15898},
year = {2026},
}