metadata
license: apache-2.0
pipeline_tag: robotics
APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
This repository contains the checkpoints for APT, a two-stage training method for Vision-Language-Action (VLA) models that emphasizes Action Expert Pretraining to improve generalization to out-of-distribution (OOD) instructions.
- Paper: APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
- Project Page: https://xukechun.github.io/papers/APT/
- Code/Github: https://github.com/xukechun/APT
Method Overview
APT factorizes the VLA policy into a Vision-Action (VA) prior and a language-conditioned VLA likelihood. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM to bypass language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior.
Sample Usage
For local inference, you can instantiate the planner directly:
from apt.infer.planner import TrajPlanner
planner = TrajPlanner(
ckpt_path="checkpoints/APT/ft_vla/ckpt_latest.pt",
device="cuda:0",
ensemble=4,
use_ema=False,
)
planner.set_prompt("Pick up the grape and place it on the pink box.")
planner.add_obs_frame(obs_frame)
actions = planner.get_action()
Citation
@article{xu2026apt,
title={APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies},
author={Xu, Kechun and Zhu, Zhenjie and Chen, Anzhe and Xiong, Rong and Wang, Yue},
journal={arXiv preprint arXiv:2606.12366},
year={2026}
}