README.md

metadata

title: Orbis 2 — Hierarchical World Model Rollout Demo
emoji: 🚗
colorFrom: indigo
colorTo: gray
sdk: gradio
sdk_version: 5.9.1
app_file: app2.py
pinned: false
license: mit

Orbis 2: A Hierarchical World Model for Driving

Upload a short driving mp4 → the model takes the tail of the clip as context → rolls out several plausible futures → returns them as videos.

This Space runs Orbis 2, the hierarchical L1–L2 model (app2.py). The previous single-level Orbis 1 demo is still in the repo as app.py; switch app_file above to serve it instead. A Space runs exactly one entrypoint.

How it works

Orbis 2 splits prediction across two levels of abstraction:

• L2 — abstract predictor. Sees a long, low-rate context window and forecasts a future scene state in latent space (long-horizon dynamics, steering).
• L1 — detail predictor. Conditioned on the L2 forecast, generates fine-grained high-rate frames, so local fidelity stays grounded in long-range context.

A shared tokenizer encodes frames to (and decodes them from) latents.

Pipeline. The uploaded clip is re-encoded to a constant 10 fps — the rollout script requires the video's native frame rate to be an exact integer multiple of the L1/L2 sampling rates. It then samples the L1 window at 10 Hz and the L2 window further back at the frozen L2 predictor's own trained rate (2 Hz), resizing to 288×512. Frames are generated autoregressively; NUM_VIDEOS futures are rolled out from the same context in a single minibatch, each with independently drawn initial noise, so they diverge.

Note the clip must be long enough to cover the L2 look-back window — short clips are rejected with an explicit error.

Models

The three checkpoints (~20 GB total) are not in this repo. They live on the Hub at sud0301/orbis2_test, laid out as:

L1/   config_distill.yaml  +  checkpoints/last.ckpt   # detail predictor
L2/   config.yaml          +  checkpoints/last.ckpt   # abstract predictor (frozen)
tok/  config.yaml          +  checkpoints/last.ckpt   # tokenizer (shared by L1 and L2)

app2.py downloads them at startup into the HF cache and exports $ORBIS2_MODELS_DIR, which L1's and L2's configs expand to locate their dependencies (L1→tok, L1→L2, L2→tok). It uses the cache path directly rather than copying into the Space, so the weights are stored once.

Setup

1. Persistent storage. Enable it in Space settings so the ~20 GB download lands on /data and survives restarts. Without it the cache is ephemeral and the models are re-fetched on every cold start.

2. HF_TOKEN secret. Required while sud0301/orbis2_test is a private repo — otherwise the startup download 401s. Add it under Settings → Variables and secrets; huggingface_hub picks it up automatically. Not needed if the model repo is public.

3. Hardware. ZeroGPU works for short rollouts (keep Rollout steps low; the @spaces.GPU budget is 300 s). Use an L40S/A100 for long-horizon rollouts.

Environment variables

Variable	Default	Purpose
ORBIS2_HF_REPO	sud0301/orbis2_test	Hub repo holding L1/, L2/, tok/
ORBIS2_MODELS_DIR	./models	Use a local model tree instead of downloading
HF_TOKEN	—	Needed while the model repo is private

Point ORBIS2_MODELS_DIR at a complete local tree to skip the Hub entirely — useful for running against a training run on a GPU box.

Controls

• Rollout steps — autoregressive steps. Each emits num_pred_frames (5) image frames, so the output clip is longer than the step count.
• Diffusion steps — flow-matching solver steps per frame.
• Seed (Advanced) — -1 draws a fresh seed each run; set it to reproduce a run.
• Eta (Advanced) — 0 is the deterministic ODE; >0 injects noise per step (SDE).