README.md

---
license: apache-2.0
tags:
- video-generation
- video-diffusion
- 3d-control
- camera-control
- image-to-video
pipeline_tag: image-to-video
library_name: diffusers
base_model:
- Wan-AI/Wan2.2-Animate-14B
---

# Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion

[![Project Page](https://img.shields.io/badge/Project-Page-yellow?logo=data:image/svg%2Bxml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAyNCAyNCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJ5ZWxsb3ciIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLWxpbmVjYXA9InJvdW5kIiBzdHJva2UtbGluZWpvaW49InJvdW5kIj48Y2lyY2xlIGN4PSIxMiIgY3k9IjEyIiByPSIxMCIvPjxsaW5lIHgxPSIyIiB5MT0iMTIiIHgyPSIyMiIgeTI9IjEyIi8+PHBhdGggZD0iTTEyIDJhMTUuMyAxNS4zIDAgMCAxIDQgMTAgMTUuMyAxNS4zIDAgMCAxLTQgMTAgMTUuMyAxNS4zIDAgMCAxLTQtMTAgMTUuMyAxNS4zIDAgMCAxIDQtMTB6Ii8+PC9zdmc+)](https://jiayi-wu-leo.github.io/real2sam2real/)
[![Paper](https://img.shields.io/badge/Paper-arXiv-b31b1b?logo=arxiv&logoColor=red)](https://arxiv.org/abs/2606.00299)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Real2SAM2Real-blue)](https://huggingface.co/JiayiWuLeo/Real2SAM2Real)
[![Code](https://img.shields.io/badge/Code-GitHub-black?logo=github)](https://github.com/jiayi-wu-umd/Real2SAM2Real)
[![Video](https://img.shields.io/badge/Video-YouTube-red?logo=youtube&logoColor=red)](https://www.youtube.com/watch?v=yWS7gLoLiXM)

**[Jiayi Wu](https://jiayi-wu-leo.github.io/)\*, [Haoming Cai](https://www.hm-cai.com/)\*, [Cornelia Fermuller](https://robotics.umd.edu/clark/faculty/1168/Cornelia-Ferm%C3%BCller), [Christopher Metzler](https://www.cs.umd.edu/people/metzler), [Yiannis Aloimonos](https://robotics.umd.edu/clark/faculty/350/Yiannis-Aloimonos)**

University of Maryland, College Park  ·  \*Equal contribution

<video controls muted loop playsinline width="100%">
  <source src="https://huggingface.co/JiayiWuLeo/Real2SAM2Real/resolve/main/assets/teaser.mp4" type="video/mp4">
</video>

## TL;DR

**Real2SAM2Real** is a 3D-aware video generation framework that integrates a generative 3D cache to provide video diffusion models (VDMs) with instance-complete geometric guidance. This enables precise, decoupled control over both camera trajectories and multi-entity motions, preventing structural collapse under complex camera shifts and severe occlusions. By fully decoupling geometry and appearance conditions, it remains robust even for non-Lambertian surfaces, fluids, and other complex phenomena.

## Overview

![Teaser figure](assets/teaser_figure.png)

Real2SAM2Real is a 3D controllable video generation framework featuring an explicitly editable 3D cache that enables precise control over both cameras and scenes. Existing methods predominantly rely on implicit diffusion priors to generate unobserved regions, which often leads to structural collapse during high-dynamic movements or complex occlusions. Our framework leverages 3D lifting models (e.g., SAM3D) to extract an explicitly editable 3D cache that serves as a robust geometric scaffold for the VDM. By capturing the entire 3D volume of foreground entities rather than just their visible shells, this cache injects holistic spatial priors into the VDM and provides dependable 3D-aware guidance for complex scene dynamics.

To effectively leverage this 3D guidance while preserving pre-trained priors, we design a **Soft Spatial-Aligned Injection** mechanism together with a minimally invasive fine-tuning strategy tailored for VDMs. We further employ masked normal maps as a cross-modal bridge to construct a **3D-free data curation and perturbation pipeline**. Extensive experiments show that Real2SAM2Real enables precise, decoupled control over camera trajectories and multi-entity motions, maintains spatiotemporal consistency under large camera shifts and severe occlusions, and eliminates perspective ambiguities caused by structural holes, erroneous facades, reflections, and refractions.

## Model Files

This repository contains the inference checkpoint used to reproduce the results shown on the project page.

```
checkpoint-infer/
├── lora_diffusion_pytorch_model.safetensors                        # LoRA weights (diffusers)
├── lora_diffusion_pytorch_model_compatible_with_comfyui.safetensors # LoRA weights (ComfyUI-compatible)
└── pose_patch_embedding.safetensors                                 # Pose patch embedding module
```

Refer to the [code repository](https://github.com/jiayi-wu-umd/Real2SAM2Real) for loading and inference instructions.

## Citation

```bibtex
@article{wu2025real2sam2real,
  title   = {Real2SAM2Real: Generative 3D Caches as Complementary Context for Video Diffusion},
  author  = {Wu, Jiayi and Cai, Haoming and Fermuller, Cornelia and Metzler, Christopher and Aloimonos, Yiannis},
  journal = {arXiv preprint arXiv:2606.00299},
  year    = {2025}
}
```