README.md

---
license: mit
pipeline_tag: image-to-video
---

# Orbis 2: A Hierarchical World Model for Driving

<table>
  <tr>
    <td><span style="font-weight: bold; font-size: 16px;">Paper:</span></td>
    <td><a href="https://arxiv.org/abs/2607.15898">https://arxiv.org/abs/2607.15898</a></td>
  </tr>
  <tr>
    <td><span style="font-weight: bold; font-size: 16px;">Project Page:</span></td>
    <td><a href="https://lmb-freiburg.github.io/orbis2.github.io/">https://lmb-freiburg.github.io/orbis2.github.io/</a></td>
  </tr>
  <tr>
    <td><span style="font-weight: bold; font-size: 16px;">Code:</span></td>
    <td><a href="https://github.com/lmb-freiburg/orbis2">https://github.com/lmb-freiburg/orbis2</a></td>
  </tr>
  <tr>
    <td><span style="font-weight: bold; font-size: 16px;">Demo:</span></td>
    <td><a href="https://huggingface.co/spaces/sud0301/orbis2_test">https://huggingface.co/spaces/sud0301/orbis2_test</a></td>
  </tr>
</table>

[Sudhanshu Mittal*](https://lmb.informatik.uni-freiburg.de/people/mittal/), [Arian Mousakhan*](https://lmb.informatik.uni-freiburg.de/people/mousakha/), [Silvio Galesso*](https://lmb.informatik.uni-freiburg.de/people/galessos/), [Karim Farid](https://lmb.informatik.uni-freiburg.de/people/faridk/), [Johannes Dienert](https://lmb.informatik.uni-freiburg.de/people/dienertj/), [Rajat Sahay](https://lmb.informatik.uni-freiburg.de/people/sahayr/), [Thomas Brox](https://lmb.informatik.uni-freiburg.de/people/brox/index.html)
<br>University of Freiburg<br>
<sub>* Main contributors</sub>

---

Orbis-2 is a hierarchical driving world model that generates long-horizon future video conditioned on past frames and an optional **steering signal**. A frozen low-frame-rate **L2** predictor provides abstract long-range context, while the **L1** detail predictor autoregressively generates high-frame-rate future frames. Steering can be given either as raw ego-motion values (speed and yaw rate) or as a 2D trajectory that the model should follow.

![Teaser](https://github.com/lmb-freiburg/orbis2/raw/main/imgs/Framework.png)

## Installation
```bash
git clone https://github.com/lmb-freiburg/orbis2.git
cd orbis2
conda env create -f environment.yml
conda activate orbis2_env
```

## Video Generation (Roll-out)
`evaluate/rollout_demo_v2.py` rolls out the world model from a single input video: it samples the L1 (high-rate) and L2 (low-rate, further back in time) context windows directly from the video, then autoregressively generates future frames.

Set the environment variable `ORBIS2_MODELS_DIR` with the path of the checkpoints folder, e.g.:
```bash
export ORBIS2_MODELS_DIR=./orbis2
```

To roll out the model using an input context video:
```bash
python evaluate/rollout_demo_v2.py \
    --video /path/to/input_video.mp4 \
    --output_dir rollout
```

To roll out with trajectory steering and an ego-centric trajectory overlay, specify an input trajectory file:
```bash
python evaluate/rollout_demo_v2.py \
    --video /path/to/input_video.mp4 \
    --trajectory_file trajectory.csv \
    --vis_mode trajectory_ego \
    --output_dir rollout_traj
```

## BibTeX
```bibtex
@article{orbis2_2026,
  author    = {Mittal, Sudhanshu and Mousakhan, Arian and Galesso, Silvio and
               Farid, Karim and Dienert, Johannes and Sahay, Rajat and Brox, Thomas},
  title     = {Orbis 2: A Hierarchical World Model for Driving},
  journal   = {arXiv preprint arXiv:2607.15898},
  year      = {2026},
}
```