X-Humanoid/Pelican-VLA05

Pelican-VLA 0.5: Attending Before Acting Benefits Generalization

Zero-shot action-pathway attention. Before any task-specific fine-tuning, Pelican-VLA 0.5 concentrates attention on the instruction-relevant object and its contact region, while other open-source VLA models attend diffusely over the robot arm, background, and irrelevant objects.

Model Summary

Pelican-VLA 0.5 is a unified Vision-Language-Action (VLA) model that integrates vision-language understanding, future-frame generation, and action prediction within a single shared Qwen3-VL 4B backbone. It is developed by the WFM System Group at the Beijing Innovation Center of Humanoid Robotics (X-Humanoid).

This repository hosts the cross-embodiment pre-trained checkpoint. It is trained on ~2,400 hours of heterogeneous manipulation data and is the model used for the zero-shot attention and zero-shot RoboTwin demonstrations in the paper. For the RoboTwin-tuned policy, see X-Humanoid/Pelican-VLA05-Robotwin.

The central design is a compact set of learnable bottleneck tokens (BotTokens) inserted between perception and action. Rather than letting the action pathway attend directly to dense visual tokens, the BotTokens form a fixed-capacity interface that compresses and routes only manipulation-relevant information into action generation. This information bottleneck induces attention-level generalization: without object annotations, segmentation masks, attention supervision, or task-specific fine-tuning, the action pathway already focuses on the instruction-relevant object and its contact region — and this behavior transfers to unseen scenes, unseen objects, and unseen robot embodiments.

Architecture

A single Qwen3-VL 4B Transformer processes one token sequence with four consecutive segments:

1. Vision-Language Prefix — multi-view images and the language instruction (scene semantics).
2. Temporal / World Middle Tokens — visual history encoded by a frozen NVIDIA Cosmos image tokenizer; supplies pixel-level dynamics and carries future-frame prediction.
3. BotTokens — 32 learnable bottleneck tokens that compress and organize manipulation-relevant information.
4. Action Suffix — robot proprioceptive state plus noisy action tokens used by the flow-matching action head to produce action trajectories.

Crucially, the action suffix cannot bypass the BotTokens to read dense visual tokens directly. Three mechanisms make the bottleneck hold: a curriculum bottleneck attention mask, an orthogonality regularizer across BotTokens, and BotToken-gated future-frame generation.

Property	Value
Backbone	Qwen3-VL 4B (Instruct)
Total parameters	~5B
Action head	Flow matching (10 denoising steps)
Bottleneck tokens	32
Action chunk size	50
Max state / action dim	32 (shorter vectors are padded)
Image resolution	224 × 224
Default dtype	bfloat16
Visual history / future-frame branch	frozen NVIDIA Cosmos-Tokenizer-CI8x8

Intended Use

• Research on generalizable vision-language-action policies and manipulation-centric attention.
• Zero-shot / few-shot deployment and analysis on robot manipulation tasks.
• A starting point for downstream fine-tuning on your own robot data.

This is an intermediate research model, not a production controller. See Limitations.

Requirements

To run this checkpoint you also need:

• Qwen3-VL-4B-Instruct weights (backbone initialization) — Qwen, Apache-2.0.
• NVIDIA Cosmos-Tokenizer-CI8x8 (frozen visual-history / future-frame branch) — nvidia/Cosmos-0.1-Tokenizer-CI8x8, NVIDIA Open Model License.
• The inference release from Open-X-Humanoid/Pelican-VLA05 (pelican_vla0.5_infer/).

How to Use

Download the checkpoint and install the inference package:

huggingface-cli download X-Humanoid/Pelican-VLA05 --local-dir ./pretrained_model

git clone https://github.com/Open-X-Humanoid/Pelican-VLA05.git
cd Pelican-VLA05/pelican_vla0.5_infer
pip install -r requirements.txt   # or: pip install -e .

export QWEN3_VL_PATH=/path/to/Qwen3-VL-4B-Instruct
# Optional (for offline Cosmos tokenizer):
# export COSMOS_TOKENIZER_PATH=/path/to/Cosmos-0.1-Tokenizer-CI8x8

For the inference API and a runnable example, see the inference release README.

Checkpoint layout

from_pretrained accepts a directory containing either a single model.safetensors or the split files backbone.safetensors + heads.safetensors (+ optional action_head.safetensors / gen_head.safetensors), together with config.json and stats.json. Loading is strict by default, so mismatched or stale parameter names fail immediately.

Attention visualization

The release ships a visualization tool that loads this checkpoint and dumps action→image attention as .npz per camera, so you can verify whether attention focuses on the task-relevant object, the gripper, and the actionable region. It runs on the shipped datasets and on your own data, provided it is packaged as a LeRobot 3.0 dataset.

Performance

Zero-shot on RoboTwin 2.0 (RoboTwin data was not used during pre-training). Deployed directly on held-out tasks with unseen objects, unseen scene layouts, and a new robot embodiment, the pre-trained policy reaches toward the correct instruction-relevant object and produces coherent, goal-directed motions.

After fine-tuning on RoboTwin 2.0 (see the RoboTwin checkpoint), Pelican-VLA 0.5 achieves the best average success rate among open-source VLA baselines, with only a 0.4-point gap between clean and randomized settings:

Method	Clean	Randomized	Average
π₀	80.0	79.5	79.8
π₀.₅	86.8	87.0	86.9
X-VLA	72.9	72.8	72.9
StarVLA-OFT	88.2	88.3	88.3
ABot-M0	86.1	85.1	85.6
LingBot-VLA	88.6	86.7	87.7
Qwen-VLA	86.1	87.2	86.7
JoyAI-RA	90.5	89.3	89.9
Hy-VLA	90.9	90.1	90.5
Pelican-VLA 0.5	91.4	91.0	91.2

Limitations

Pelican-VLA 0.5 is an intermediate stage toward practical zero-shot manipulation. A representation-to-action gap remains: the model begins to identify what to act upon, but reliable execution (stable grasping, contact timing, precise placement) is not solved. This is attributed mainly to:

• Limited action experience — the current checkpoint sees only ~2,400 hours of heterogeneous data with some language-action noise.
• Joint-position actions — more embodiment-specific than end-effector pose representations, which limits cross-embodiment transfer.

A stronger version trained on approximately 7,000 hours, with improved action parameterization and stricter data curation, is planned to supersede this checkpoint.

License

Released under the Apache-2.0 License. This license covers the source code and this checkpoint distribution. Component model assets keep their own licenses: Qwen3-VL weights are Apache-2.0; NVIDIA Cosmos tokenizer weights use the NVIDIA Open Model License. Third-party components (LeRobot, openpi, Hugging Face Transformers, NVIDIA Cosmos) are attributed in the release's NOTICE and THIRD_PARTY_LICENSES.md.

Citation

@article{ding2026pelican,
  title   = {Pelican-VLA 0.5: Attending Before Acting Benefits Generalization},
  author  = {Ding, Zeyuan and Liu, Wenhai and Xu, Yang and Hu, Jiayu and Chen, Yinda and
             Zhang, Yi and Dai, Yong and Tang, Jian and Ju, Xiaozhu},
  journal = {arXiv preprint arXiv:2607.06655},
  year    = {2026},
  url     = {https://arxiv.org/abs/2607.06655}
}

Acknowledgement

We thank the developers of LeRobot, openpi, Qwen3-VL, and NVIDIA Cosmos for their contributions to the open-source community.