File size: 3,446 Bytes
e2451af b75f837 9417d3c b75f837 9417d3c 63fdc9f 9417d3c b75f837 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 | ---
license: mit
pipeline_tag: image-to-video
---
# Orbis 2: A Hierarchical World Model for Driving
<table>
<tr>
<td><span style="font-weight: bold; font-size: 16px;">Paper:</span></td>
<td><a href="https://arxiv.org/abs/2607.15898">https://arxiv.org/abs/2607.15898</a></td>
</tr>
<tr>
<td><span style="font-weight: bold; font-size: 16px;">Project Page:</span></td>
<td><a href="https://lmb-freiburg.github.io/orbis2.github.io/">https://lmb-freiburg.github.io/orbis2.github.io/</a></td>
</tr>
<tr>
<td><span style="font-weight: bold; font-size: 16px;">Code:</span></td>
<td><a href="https://github.com/lmb-freiburg/orbis2">https://github.com/lmb-freiburg/orbis2</a></td>
</tr>
<tr>
<td><span style="font-weight: bold; font-size: 16px;">Demo:</span></td>
<td><a href="https://huggingface.co/spaces/sud0301/orbis2_test">https://huggingface.co/spaces/sud0301/orbis2_test</a></td>
</tr>
</table>
[Sudhanshu Mittal*](https://lmb.informatik.uni-freiburg.de/people/mittal/), [Arian Mousakhan*](https://lmb.informatik.uni-freiburg.de/people/mousakha/), [Silvio Galesso*](https://lmb.informatik.uni-freiburg.de/people/galessos/), [Karim Farid](https://lmb.informatik.uni-freiburg.de/people/faridk/), [Johannes Dienert](https://lmb.informatik.uni-freiburg.de/people/dienertj/), [Rajat Sahay](https://lmb.informatik.uni-freiburg.de/people/sahayr/), [Thomas Brox](https://lmb.informatik.uni-freiburg.de/people/brox/index.html)
<br>University of Freiburg<br>
<sub>* Main contributors</sub>
---
Orbis-2 is a hierarchical driving world model that generates long-horizon future video conditioned on past frames and an optional **steering signal**. A frozen low-frame-rate **L2** predictor provides abstract long-range context, while the **L1** detail predictor autoregressively generates high-frame-rate future frames. Steering can be given either as raw ego-motion values (speed and yaw rate) or as a 2D trajectory that the model should follow.
## Installation
```bash
git clone https://github.com/lmb-freiburg/orbis2.git
cd orbis2
conda env create -f environment.yml
conda activate orbis2_env
```
## Video Generation (Roll-out)
`evaluate/rollout_demo_v2.py` rolls out the world model from a single input video: it samples the L1 (high-rate) and L2 (low-rate, further back in time) context windows directly from the video, then autoregressively generates future frames.
Set the environment variable `ORBIS2_MODELS_DIR` with the path of the checkpoints folder, e.g.:
```bash
export ORBIS2_MODELS_DIR=./orbis2
```
To roll out the model using an input context video:
```bash
python evaluate/rollout_demo_v2.py \
--video /path/to/input_video.mp4 \
--output_dir rollout
```
To roll out with trajectory steering and an ego-centric trajectory overlay, specify an input trajectory file:
```bash
python evaluate/rollout_demo_v2.py \
--video /path/to/input_video.mp4 \
--trajectory_file trajectory.csv \
--vis_mode trajectory_ego \
--output_dir rollout_traj
```
## BibTeX
```bibtex
@article{orbis2_2026,
author = {Mittal, Sudhanshu and Mousakhan, Arian and Galesso, Silvio and
Farid, Karim and Dienert, Johannes and Sahay, Rajat and Brox, Thomas},
title = {Orbis 2: A Hierarchical World Model for Driving},
journal = {arXiv preprint arXiv:2607.15898},
year = {2026},
}
``` |