README.md

metadata

language:
  - en
license: mit
pipeline_tag: robotics
library_name: lam

villa-X: A Vision-Language-Latent-Action Model

   

This is the official Hugging Face repository for villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models.

Abstract

Visual-Language-Action (VLA) models have emerged as a popular paradigm for learning robot manipulation policies that can follow language instructions and generalize to novel scenarios. Recent work has begun to explore the incorporation of latent actions, an abstract representation of visual change between two frames, into VLA pre-training. In this paper, we introduce villa-X, a novel Visual-Language-Latent-Action (ViLLA) framework that advances latent action modeling for learning generalizable robot manipulation policies. Our approach improves both how latent actions are learned and how they are incorporated into VLA pre-training. Together, these contributions enable villa-X to achieve superior performance across simulated environments including SIMPLER and LIBERO, as well as on two real-world robot setups including gripper and dexterous hand manipulation. We believe the ViLLA paradigm holds significant promise, and that our villa-X provides a strong foundation for future research.

Overview

• We improve latent action learning by introducing an extra proprio FDM, which aligns latent tokens with underlying robot states and actions and grounds them in physical dynamics.
• We propose to jointly learn a latent action expert and a robot action expert through joint diffusion in the policy model, conditioning robot action prediction on latent actions to fully exploit their potential.
• Our method demonstrates superior performance on simulated environments as well as on real-world robotic tasks. The latent action expert can effectively plan into future with both visual and proprio state planning.

Usage

Setup

1. Clone the repository.

   git clone https://github.com/microsoft/villa-x.git
   cd villa-x
   

2. Install the required packages.

   sudo apt-get install -y build-essential zlib1g-dev libffi-dev libssl-dev libbz2-dev libreadline-dev libsqlite3-dev liblzma-dev libncurses-dev tk-dev python3-dev ffmpeg -y
   curl -LsSf https://astral.sh/uv/install.sh | sh # Skip this step if you already have uv installed
   uv sync
   

Inference with Pre-trained Latent Action Model

1. Download the pre-trained models from Hugging Face.

2. Load the latent action model.

   from lam import IgorModel
   
   lam = IgorModel.from_pretrained("LOCAL_MODEL_DIRECTORY").cuda()
   

3. Extract the latent actions from a video.

   def read_video(fp: str):
       from torchvision.io import read_video
   
       video, *_ = read_video(fp, pts_unit="sec")
       return video
   
   video = read_video("path/to/video.mp4").cuda()  # Load your video here
   latent_action = lam.idm(video)
   

4. Use image FDM to reconstruct future frames from the latent actions.

   frames = []
   for i in range(len(latent_action[0])):
       pred = lam.apply_latent_action(video[i], latent_action[0][i])
       frames.append(pred)
   

We also provide a Jupyter notebook for a step-by-step guide on how to use the pre-trained latent action model.

Pre-trained Models

Model ID	Description	Params	Link
microsoft/villa-x/lam	Latent action model	955M	🤗 Link

Citation

@article{chen2025villa0x0,
  title   = {villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models},
  author  = {Xiaoyu Chen and Hangxing Wei and Pushi Zhang and Chuheng Zhang and Kaixin Wang and Yanjiang Guo and Rushuai Yang and Yucen Wang and Xinquan Xiao and Li Zhao and Jianyu Chen and Jiang Bian},
  year    = {2025},
  journal = {arXiv preprint arXiv: 2507.23682}
}

Credits

We are grateful for the open-source projects like Open Sora, taming-transformers, open-pi-zero, MAE and timm. Their contributions have been invaluable in the development of villa-X.