| --- |
| license: apache-2.0 |
| pipeline_tag: robotics |
| --- |
| |
| # APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies |
|
|
| This repository contains the checkpoints for APT, a two-stage training method for Vision-Language-Action (VLA) models that emphasizes Action Expert Pretraining to improve generalization to out-of-distribution (OOD) instructions. |
|
|
| * **Paper**: [APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies](https://huggingface.co/papers/2606.12366) |
| * **Project Page**: [https://xukechun.github.io/papers/APT/](https://xukechun.github.io/papers/APT/) |
| * **Code/Github**: [https://github.com/xukechun/APT](https://github.com/xukechun/APT) |
|
|
| ## Method Overview |
|
|
| APT factorizes the VLA policy into a Vision-Action (VA) prior and a language-conditioned VLA likelihood. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM to bypass language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. |
|
|
| ## Sample Usage |
|
|
| For local inference, you can instantiate the planner directly: |
|
|
| ```python |
| from apt.infer.planner import TrajPlanner |
| |
| planner = TrajPlanner( |
| ckpt_path="checkpoints/APT/ft_vla/ckpt_latest.pt", |
| device="cuda:0", |
| ensemble=4, |
| use_ema=False, |
| ) |
| planner.set_prompt("Pick up the grape and place it on the pink box.") |
| planner.add_obs_frame(obs_frame) |
| actions = planner.get_action() |
| ``` |
|
|
| ## Citation |
|
|
| ```bibtex |
| @article{xu2026apt, |
| title={APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies}, |
| author={Xu, Kechun and Zhu, Zhenjie and Chen, Anzhe and Xiong, Rong and Wang, Yue}, |
| journal={arXiv preprint arXiv:2606.12366}, |
| year={2026} |
| } |
| ``` |
