File size: 3,106 Bytes
2bf05b6 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 | ---
license: apache-2.0
language:
- en
tags:
- robotics
- 3d-detection
- vision-language-action
- pose-estimation
- grounding
library_name: transformers
pipeline_tag: robotics
---
<div align="center">
# PoseVLA Stage-1: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies
[](https://arxiv.org/abs/2602.19710)
[](https://hetolin.github.io/PoseVLA/)
[](https://github.com/hetolin/PoseVLA)
</div>
## Model Description
**PoseVLA** is a Vision-Language-Action (VLA) model that leverages universal 3D pose pretraining for generalizable robotic manipulation. This checkpoint is the **Stage-1 pretrained model**, jointly trained on large-scale 3D detection and robot action data.
- **Architecture**: PaliGemma-3B + Action Expert (π0-based, trained from scratch) with Flow Matching
- **Pretraining Data**: Omni3D, Omni6D, BOP, GraspClutter6D (3D tasks) + Agibot, InternData-A1 (robot actions)
- **Input**: Multi-view RGB images + Depth priors + Camera intrinsics
- **Output**: 3D object detection (Next-Token Prediction) / Robot actions (Flow Matching)
## Usage
### 3D Grounding Inference
```python
from posevla.modeling_posevla import PoseVLAPolicy, PoseVLAConfig, bin_tokenizer
from utils.mapping_token import decode_text_to_scene_with_tokenizer
# Load model
policy = PoseVLAPolicy.from_pretrained("hetolin/PoseVLA-stage1", local_files_only=False, config=posevla_config)
policy = policy.eval().to(torch.bfloat16).cuda()
# Inference
output_res = policy.forward_evaluate_ntp(batch)
pred_text = output_res["pred"][0]
pred_res = decode_text_to_scene_with_tokenizer(pred_text, bin_tokenizer)
```
### Full Inference Script
See [`infer_grounding3d.py`](https://github.com/hetolin/PoseVLA/blob/main/infer_grounding3d.py) for complete real-world RGB-D inference pipeline.
## Training Details
| Hyperparameter | Value |
|:---|:---|
| Base model | PaliGemma-3B-pt-224 |
| Action Expert | π0 (Flow Matching, from scratch) |
| Image resolution | 224 × 224 |
| Optimizer | AdamW |
| Learning rate | 5e-5 |
| Weight decay | 1e-10 |
| Precision | bf16 |
| GPUs | 16 × H20 |
| Batch size | 7 per GPU |
| Training steps | 100K |
## Intended Use
- **3D Object Grounding**: Open-vocabulary 3D detection from RGB-D images
- **Robot Manipulation**: Pretrained backbone for downstream robotic fine-tuning (e.g., RoboTwin)
- **Research**: Studying the synergy between 3D spatial understanding and robot action learning
## Citation
```bibtex
@article{lin2026posevla,
title={PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies},
author={Lin, Haitao and Yu, Hanyang and Huang, Jingshun and Zhang, He and Ling, Yonggen and Tan, Ping and Xue, Xiangyang and Fu, Yanwei},
journal={arXiv preprint arXiv:2602.19710},
year={2026}
}
```
## License
This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0). |