README.md

---
title: Orbis 2 — Hierarchical World Model Rollout Demo
emoji: 🚗
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: "5.9.1"
app_file: app2.py
pinned: false
license: mit
---

# Orbis 2: A Hierarchical World Model for Driving

Upload a short driving mp4 → the model takes the tail of the clip as context →
rolls out several plausible futures → returns them as videos.

This Space runs **Orbis 2**, the hierarchical L1–L2 model (`app2.py`). The previous
single-level Orbis 1 demo is still in the repo as `app.py`; switch `app_file` above
to serve it instead. A Space runs exactly one entrypoint.

## How it works

Orbis 2 splits prediction across two levels of abstraction:

- **L2 — abstract predictor.** Sees a long, low-rate context window and forecasts a
  future scene state in latent space (long-horizon dynamics, steering).
- **L1 — detail predictor.** Conditioned on the L2 forecast, generates fine-grained
  high-rate frames, so local fidelity stays grounded in long-range context.

A shared **tokenizer** encodes frames to (and decodes them from) latents.

**Pipeline.** The uploaded clip is re-encoded to a constant 10 fps — the rollout
script requires the video's native frame rate to be an exact integer multiple of the
L1/L2 sampling rates. It then samples the L1 window at 10 Hz and the L2 window
further back at the frozen L2 predictor's own trained rate (2 Hz), resizing to
288×512. Frames are generated autoregressively; `NUM_VIDEOS` futures are rolled out
from the same context **in a single minibatch**, each with independently drawn
initial noise, so they diverge.

Note the clip must be long enough to cover the L2 look-back window — short clips are
rejected with an explicit error.

## Models

The three checkpoints (~20 GB total) are **not in this repo**. They live on the Hub
at [`sud0301/orbis2_test`](https://huggingface.co/sud0301/orbis2_test), laid out as:

```
L1/   config_distill.yaml  +  checkpoints/last.ckpt   # detail predictor
L2/   config.yaml          +  checkpoints/last.ckpt   # abstract predictor (frozen)
tok/  config.yaml          +  checkpoints/last.ckpt   # tokenizer (shared by L1 and L2)
```

`app2.py` downloads them at startup into the HF cache and exports
`$ORBIS2_MODELS_DIR`, which L1's and L2's configs expand to locate their
dependencies (L1→tok, L1→L2, L2→tok). It uses the cache path directly rather than
copying into the Space, so the weights are stored once.

## Setup

1. **Persistent storage.** Enable it in Space settings so the ~20 GB download lands
   on `/data` and survives restarts. Without it the cache is ephemeral and the models
   are re-fetched on every cold start.

2. **`HF_TOKEN` secret.** Required while `sud0301/orbis2_test` is a **private** repo —
   otherwise the startup download 401s. Add it under Settings → Variables and secrets;
   `huggingface_hub` picks it up automatically. Not needed if the model repo is public.

3. **Hardware.** ZeroGPU works for short rollouts (keep *Rollout steps* low; the
   `@spaces.GPU` budget is 300 s). Use an L40S/A100 for long-horizon rollouts.

### Environment variables

| Variable | Default | Purpose |
|---|---|---|
| `ORBIS2_HF_REPO` | `sud0301/orbis2_test` | Hub repo holding `L1/`, `L2/`, `tok/` |
| `ORBIS2_MODELS_DIR` | `./models` | Use a local model tree instead of downloading |
| `HF_TOKEN` | — | Needed while the model repo is private |

Point `ORBIS2_MODELS_DIR` at a complete local tree to skip the Hub entirely — useful
for running against a training run on a GPU box.

## Controls

- **Rollout steps** — autoregressive steps. Each emits `num_pred_frames` (5) image
  frames, so the output clip is longer than the step count.
- **Diffusion steps** — flow-matching solver steps per frame.
- **Seed** (Advanced) — `-1` draws a fresh seed each run; set it to reproduce a run.
- **Eta** (Advanced) — `0` is the deterministic ODE; `>0` injects noise per step (SDE).