[CVPR 2026] RayNova: Scale-Temporal Autoregressive World Modeling in Ray Space

Project Page | Paper | Code

World foundation models aim to simulate the evolution of the real world with physically plausible behavior. Unlike prior methods that handle spatial and temporal correlations separately, RAYNOVA is a geometry-agnostic multiview world model for driving scenarios that employs a dual-causal autoregressive framework. It follows both scale-wise and temporal topological orders in the autoregressive process, and leverages global attention for unified 4D spatio-temporal reasoning. Different from existing works that impose strong 3D geometric priors, RAYNOVA constructs an isotropic spatio-temporal representation across views, frames, and scales based on relative Plücker-ray positional encoding, enabling robust generalization to diverse camera setups and ego motions.

Model

This release is the RayNova 2B model (Infinity-2B backbone, depth 32, width 2048), trained exclusively on publicly available data — nuPlan and nuScenes in ScenarioNet format. Performance may be slightly lower than our internal model trained on the full dataset.

File	Description
slim-gpt_model_latest.pth	RayNova transformer weights (plain PyTorch state_dict)
config.yaml	Training configuration (defines model hyperparameters used at inference)

Companion weights (not included here, from their original releases):

• VAE: FoundationVision/Infinity infinity_vae_d32reg.pth
• Text encoder: google/flan-t5-xl

Usage

git clone https://github.com/Applied-Intuition-Open-Source/RayNova
cd RayNova
mkdir -p weights && cd weights

wget https://huggingface.co/FoundationVision/Infinity/resolve/main/infinity_vae_d32reg.pth
git clone https://huggingface.co/google/flan-t5-xl
huggingface-cli download AppliedIntuitionResearch/RayNova --local-dir raynova

cd ..

Then follow the inference notebook tools/interactive_infer.ipynb, setting model_path to weights/raynova/slim-gpt_model_latest.pth (the accompanying config.yaml is picked up from the same directory). Set enable_model_cache=False in the notebook args since this is already a slim (inference-only) state dict.

GPU memory. Inference at 384×672 (pn='0.25M') needs more than 24 GB of GPU memory (A100/H100-class recommended) — the autoregressive KV cache alone saturates a 24 GB card over a multi-window rollout. On a 24 GB card (e.g. RTX 4090), use 192×336 (pn='0.06M'), which runs out of the box with all 8 views (~20 GB peak).

Citation

@article{xie2026raynova,
  title={RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space},
  author={Xie, Yichen and Peng, Chensheng and Abdelfattah, Mazen and Hu, Yihan and Yang, Jiezhi and Higgins, Eric and Brigden, Ryan and Tomizuka, Masayoshi and Zhan, Wei},
  journal={arXiv preprint arXiv:2602.20685},
  year={2026}
}