| --- |
| license: cc-by-nc-sa-4.0 |
| tags: |
| - world-model |
| - autonomous-driving |
| - video-generation |
| - autoregressive |
| - multiview |
| paper: arxiv.org/abs/2602.20685 |
| --- |
| |
| # [CVPR 2026] RayNova: Scale-Temporal Autoregressive World Modeling in Ray Space |
|
|
| **[Project Page](https://raynova-ai.github.io/) | [Paper](https://arxiv.org/abs/2602.20685) | [Code](https://github.com/Applied-Intuition-Open-Source/RayNova)** |
|
|
| World foundation models aim to simulate the evolution of the real world with physically plausible behavior. Unlike prior methods that handle spatial and temporal correlations separately, RAYNOVA is a geometry-agnostic multiview world model for driving scenarios that employs a dual-causal autoregressive framework. It follows both scale-wise and temporal topological orders in the autoregressive process, and leverages global attention for unified 4D spatio-temporal reasoning. Different from existing works that impose strong 3D geometric priors, RAYNOVA constructs an isotropic spatio-temporal representation across views, frames, and scales based on relative Plücker-ray positional encoding, enabling robust generalization to diverse camera setups and ego motions. |
|
|
| ## Model |
|
|
| This release is the **RayNova 2B** model (Infinity-2B backbone, depth 32, width 2048), trained exclusively on **publicly available data** — nuPlan and nuScenes in [ScenarioNet](https://github.com/metadriverse/scenarionet) format. Performance may be slightly lower than our internal model trained on the full dataset. |
|
|
| | File | Description | |
| |------|-------------| |
| | `slim-gpt_model_latest.pth` | RayNova transformer weights (plain PyTorch `state_dict`) | |
| | `config.yaml` | Training configuration (defines model hyperparameters used at inference) | |
|
|
| Companion weights (not included here, from their original releases): |
|
|
| - VAE: [`FoundationVision/Infinity`](https://huggingface.co/FoundationVision/Infinity) `infinity_vae_d32reg.pth` |
| - Text encoder: [`google/flan-t5-xl`](https://huggingface.co/google/flan-t5-xl) |
|
|
| ## Usage |
|
|
| ```bash |
| git clone https://github.com/Applied-Intuition-Open-Source/RayNova |
| cd RayNova |
| mkdir -p weights && cd weights |
| |
| wget https://huggingface.co/FoundationVision/Infinity/resolve/main/infinity_vae_d32reg.pth |
| git clone https://huggingface.co/google/flan-t5-xl |
| huggingface-cli download AppliedIntuitionResearch/RayNova --local-dir raynova |
| |
| cd .. |
| ``` |
|
|
| Then follow the inference notebook [`tools/interactive_infer.ipynb`](https://github.com/Applied-Intuition-Open-Source/RayNova/blob/main/tools/interactive_infer.ipynb), setting `model_path` to `weights/raynova/slim-gpt_model_latest.pth` (the accompanying `config.yaml` is picked up from the same directory). Set `enable_model_cache=False` in the notebook args since this is already a slim (inference-only) state dict. |
|
|
| **GPU memory.** Inference at 384×672 (`pn='0.25M'`) needs more than 24 GB of GPU memory (A100/H100-class recommended) — the autoregressive KV cache alone saturates a 24 GB card over a multi-window rollout. On a 24 GB card (e.g. RTX 4090), use 192×336 (`pn='0.06M'`), which runs out of the box with all 8 views (~20 GB peak). |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{xie2026raynova, |
| title={RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space}, |
| author={Xie, Yichen and Peng, Chensheng and Abdelfattah, Mazen and Hu, Yihan and Yang, Jiezhi and Higgins, Eric and Brigden, Ryan and Tomizuka, Masayoshi and Zhan, Wei}, |
| journal={arXiv preprint arXiv:2602.20685}, |
| year={2026} |
| } |
| ``` |
|
|
