RynnWorld-4D

4D Embodied World Models for Robotic Manipulation

💫 Project Page  |   🤗 Hugging Face Collection  |   🤖 ModelScope  |   📄 Technical Report

---

Model Summary

RynnWorld-4D is a 4D embodied world model that generates synchronized RGB, depth, and optical flow (RGB-DF) videos from a single reference image and text prompt. Unlike conventional 2D pixel-prediction world models, RynnWorld-4D captures the underlying 3D geometry and temporal motion trajectories of a scene, producing a physically grounded representation that bridges generative world modeling and low-level robotic control.

This repository hosts the Stage-3 checkpoint (tri-branch full-parameter SFT with joint cross-modal attention) built on top of Wan2.2-TI2V-5B-Diffusers.

Key Highlights

• Projective 4D Representation — Unified RGB-DF format lets pixels be unprojected into metric 3D scene flow for precise geometric and kinetic grounding.
• Tri-Branch Diffusion Architecture — Three dedicated transformer branches (RGB / depth / flow) with mutual cross-modal joint attention ensure appearance, geometry, and motion evolve with high spatio-temporal consistency.
• Cosine-Decay Joint Injection — During Stage-3 training, the depth/flow → RGB injection is gradually annealed from 1.0 to 0.0, preserving cross-modal consistency while preventing RGB quality degradation.
• Action-from-Latent Policy (companion model) — The paired RynnWorld-4D-Policy consumes internal 4D latents directly for high-frequency (9 Hz+), closed-loop bimanual manipulation.

---

Model Architecture

• Backbone: Wan2.2-TI2V-5B (Diffusion Transformer, ~5B params)
• Branches: 3 parallel transformer streams — RGB, Depth, Optical Flow
• Cross-modal fusion: Joint attention with 3D RoPE, applied every 3 layers across all 30 blocks
  • --joint_start_layer 0 --joint_end_layer 30 --joint_every_n_layers 3
  • --joint_frame_wise True (attention restricted to same-frame tokens)
  • --joint_use_rope True
• Fusion mode: bidirectional joint attention with cosine-decayed video injection

---

Files

RynnWorld-4D/
├── pytorch_model/
│   └── mp_rank_00_model_states.pt   # 28 GB — full trained weights
└── ema_weights.pt                   # 28 GB — EMA shadow weights (recommended for inference)

Both files together (~56 GB) are required for the standard inference path in inference-sft.py. The EMA weights are loaded on top of the base weights and typically yield the best sample quality.

---

Quick Start

1. Environment

git clone https://github.com/Alibaba-DAMO-Academy/RynnWorld-4D
cd RynnWorld-4D

conda create -n rynnworld4d python=3.10 -y
conda activate rynnworld4d
pip install torch torchvision --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt
pip install -e . --no-build-isolation

2. Download the base backbone and this checkpoint

# base backbone (required)
huggingface-cli download Wan-AI/Wan2.2-TI2V-5B-Diffusers \
    --local-dir ./pretrained/Wan2.2-TI2V-5B-Diffusers

# RynnWorld-4D Stage-3 weights (this repo)
huggingface-cli download Alibaba-DAMO-Academy/RynnWorld-4D \
    --local-dir ./pretrained/RynnWorld-4D

3. Inference

python inference-sft.py \
    --model_path       ./pretrained/Wan2.2-TI2V-5B-Diffusers \
    --checkpoint_path  ./pretrained/RynnWorld-4D \
    --json_path        ./data/sample.json \
    --output_dir       ./results/rynnworld4d \
    --fusion_mode      joint \
    --share_ffn        False \
    --joint_start_layer 0 \
    --joint_end_layer   30 \
    --joint_every_n_layers 3 \
    --joint_frame_wise  True \
    --joint_use_rope    True \
    --joint_unidirectional False \
    --zero_fusion       False \
    --use_ema           True \
    --num_inference_steps 50 \
    --guidance_scale    1.0

Config flags must match training. The important ones for this checkpoint:

Flag	Value	Reason
--fusion_mode	joint	Stage-3 uses joint cross-modal attention.
--joint_use_rope	True	3D RoPE is enabled in joint attention.
--joint_unidirectional	False	Trained as bidirectional (with video-side cosine decay).
--use_ema	True	Loads EMA weights on top of base for best quality.
--zero_fusion	False	Fusion weights are trained; do not zero them.

Each sample produces three synchronized streams:

<output_dir>/<sample_id>/
├── rgb.mp4       # generated RGB
├── depth.mp4     # generated depth
└── flow.mp4      # generated optical flow

---

Training Recipe (Summary)

RynnWorld-4D is trained in three stages on top of the Wan2.2-TI2V-5B backbone. This checkpoint is the output of Stage 3.

Stage	Objective	Trainable	Fusion
1	Full-parameter SFT — warm up all three branches (RGB / depth / flow) independently.	All branches	None
2	Enable 3D-RoPE joint cross-modal attention; freeze non-joint params. Train bidirectional joint layers.	Joint attention only	Bidirectional, RoPE
3	Full-parameter fine-tuning with cosine-decay video injection: the depth/flow → RGB gate multiplier anneals from 1.0 → 0.0 over --joint_video_decay_steps, letting the model absorb early-stage cross-modal consistency while gradually restoring independent RGB generation.	Everything	Bidirectional → unidirectional (decayed)

See the training scripts (scripts/rynnworld4d-stage{1,2,3}.sh) in the code repository for full hyperparameters.

---

Intended Uses

• Research on 4D world models, cross-modal video generation, and geometry-aware embodied AI.
• Feature extraction for downstream robotic policy learning (see RynnWorld-4D-Policy).
• Depth / flow-consistent video synthesis conditioned on a first-frame image + text prompt.

Out-of-Scope

• Non-embodied / non-robotic content generation is possible but not the training focus.
• The model is not suitable for real-time / on-device inference without further distillation.
• Not designed for photorealistic humans, faces, or open-domain creative video generation.

---

Limitations

• Trained primarily on robotic manipulation and egocentric datasets; performance on unrelated domains (natural scenes, cinematic footage) may be limited.
• 25-frame, 480×832 latent resolution — extended sequences may drift.
• Bidirectional joint attention can occasionally propagate depth/flow artifacts to RGB, mitigated but not eliminated by the cosine-decay schedule.
• Guidance-scale > 1.0 requires a null-prompt embedding file (see inference-sft.py).

---

Companion Model

• RynnWorld-4D-Policy — a lightweight flow-matching action head trained on top of the frozen RynnWorld-4D backbone for high-frequency bimanual manipulation.

---

Acknowledgements

Built on top of and inspired by Wan2.2-TI2V-5B, Depth-Anything-3, Video Prediction Policy (VPP), and ptlflow.

---

Citation

@article{rynnworld4d,
  title  = {RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation},
  author = {DAMO Academy, Alibaba Group},
  year   = {2026},
}

---

License

Apache License 2.0. Vendored third-party code follows its original upstream licenses (preserved in third_party/*/LICENSE).