README.md

metadata

license: mit
pipeline_tag: robotics
tags:
  - robotics-simulation
  - gaussian-splatting
  - deformable-objects

Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions

Kaifeng Zhang^1,2*, Shuo Sha^1,2*, Hanxiao Jiang¹, Matthew Loper², Hyunjong Song², Guangyan Cai², Zhuo Xu³, Xiaochen Hu², Changxi Zheng^1,2, Yunzhu Li^1,2

¹Columbia University, ²SceniX Inc., ³Google DeepMind, ^*Equal contribution. Work partially done while interning at SceniX Inc.

This model is presented in the paper Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions.

Project Page: https://real2sim-eval.github.io/ Code Repository: https://github.com/kywind/real2sim-eval

Abstract

Robotic manipulation policies are advancing rapidly, but their direct evaluation in the real world remains costly, time-consuming, and difficult to reproduce, particularly for tasks involving deformable objects. Simulation provides a scalable and systematic alternative, yet existing simulators often fail to capture the coupled visual and physical complexity of soft-body interactions. We present a real-to-sim policy evaluation framework that constructs soft-body digital twins from real-world videos and renders robots, objects, and environments with photorealistic fidelity using 3D Gaussian Splatting. We validate our approach on representative deformable manipulation tasks, including plush toy packing, rope routing, and T-block pushing, demonstrating that simulated rollouts correlate strongly with real-world execution performance and reveal key behavioral patterns of learned policies. Our results suggest that combining physics-informed reconstruction with high-quality rendering enables reproducible, scalable, and accurate evaluation of robotic manipulation policies.

Overview

This repository contains the official implementation of the Real-to-Sim-Policy-Eval framework. The main components are:

• Construct simulation assets: visualize object Gaussians, articulate robot Gaussians, define and render different object layouts.
• Deploy trained PhysTwin to simulate deformable object Gaussians.
• Train policies using the policy_training submodule located in policy/.
• Evaluate policy in the constructed simulation environment.
• Interactive control of the Gaussian-based simulation.

Sample Usage

Installation

This project uses uv, a fast Python package and project manager. The installation has been tested with CUDA 12.8. Make sure to install PyTorch with the matching CUDA toolkit. GLM and FFmpeg are also required.

# create and activate a python venv
uv venv --python=3.11
source .venv/bin/activate

# install main dependencies
uv pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 --index-url https://download.pytorch.org/whl/cu128
uv pip install -r pyproject.toml

# pull the policy submodule and install dependencies
git submodule update --init --recursive
cd policy
uv pip install -r pyproject.toml

# install Gaussian Splatting rasterizer
cd ../third-party/diff-gaussian-rasterization-w-depth
uv pip install --no-build-isolation -e .

# install a modified version of urdfpy to avoid version conflicts
cd ../third-party/urdfpy-0.0.22
uv pip install -e .

# return to the project home
cd ../..

Policy Evaluation

Make sure you download the Gaussian Splatting scans, PhysTwin checkpoints, and policy checkpoints for the corresponding task as specified in the GitHub README Assets and Checkpoints section.

CUDA_VISIBLE_DEVICES=0 python experiments/eval_policy.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001 \
policy.inference_cfg_path=policy/configs/inference/insert_rope.json \
policy.checkpoint_path=log/policy_checkpoints/act-rope-routing/checkpoints/007000

Keyboard Interactive Control

CUDA_VISIBLE_DEVICES=0 python experiments/keyboard_teleop.py \
gs=rope \
env=xarm_gripper \
physics.ckpt_path=log/phystwin/rope \
physics.case_name=rope_0001

For more detailed instructions, including parallelized policy evaluation, motion replay, and policy training, please refer to the GitHub repository.

Citation

If you find this repo useful, please consider citing the paper:

@article{zhang2025real,
    title={Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions},
    author={Zhang, Kaifeng and Sha, Shuo and Jiang, Hanxiao and Loper, Matthew and Song, Hyunjong and Cai, Guangyan and Xu, Zhuo and Hu, Xiaochen and Zheng, Changxi and Li, Yunzhu},
    journal={arXiv preprint arXiv:2511.04665},
    year={2025}
}