README.md

---
license: apache-2.0
language:
- en
tags:
- robotics
- 3d-detection
- vision-language-action
- pose-estimation
- grounding
library_name: transformers
pipeline_tag: robotics
---

<div align="center">

# PoseVLA Stage-1: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

[![arXiv](https://img.shields.io/badge/arXiv-2602.19710-b31b1b.svg)](https://arxiv.org/abs/2602.19710)
[![Project Page](https://img.shields.io/badge/Project_Page-PoseVLA-2ea44f.svg)](https://hetolin.github.io/PoseVLA/)
[![GitHub](https://img.shields.io/badge/GitHub-PoseVLA-181717.svg)](https://github.com/hetolin/PoseVLA)

</div>

## Model Description

**PoseVLA** is a Vision-Language-Action (VLA) model that leverages universal 3D pose pretraining for generalizable robotic manipulation. This checkpoint is the **Stage-1 pretrained model**, jointly trained on large-scale 3D detection and robot action data.

- **Architecture**: PaliGemma-3B + Action Expert (π0-based, trained from scratch) with Flow Matching
- **Pretraining Data**: Omni3D, Omni6D, BOP, GraspClutter6D (3D tasks) + Agibot, InternData-A1 (robot actions)
- **Input**: Multi-view RGB images + Depth priors + Camera intrinsics
- **Output**: 3D object detection (Next-Token Prediction) / Robot actions (Flow Matching)

## Usage

### 3D Grounding Inference

```python
from posevla.modeling_posevla import PoseVLAPolicy, PoseVLAConfig, bin_tokenizer
from utils.mapping_token import decode_text_to_scene_with_tokenizer

# Load model
policy = PoseVLAPolicy.from_pretrained("hetolin/PoseVLA-stage1", local_files_only=False, config=posevla_config)
policy = policy.eval().to(torch.bfloat16).cuda()

# Inference
output_res = policy.forward_evaluate_ntp(batch)
pred_text = output_res["pred"][0]
pred_res = decode_text_to_scene_with_tokenizer(pred_text, bin_tokenizer)
```

### Full Inference Script

See [`infer_grounding3d.py`](https://github.com/hetolin/PoseVLA/blob/main/infer_grounding3d.py) for complete real-world RGB-D inference pipeline.

## Training Details

| Hyperparameter | Value |
|:---|:---|
| Base model | PaliGemma-3B-pt-224 |
| Action Expert | π0 (Flow Matching, from scratch) |
| Image resolution | 224 × 224 |
| Optimizer | AdamW |
| Learning rate | 5e-5 |
| Weight decay | 1e-10 |
| Precision | bf16 |
| GPUs | 16 × H20 |
| Batch size | 7 per GPU |
| Training steps | 100K |

## Intended Use

- **3D Object Grounding**: Open-vocabulary 3D detection from RGB-D images
- **Robot Manipulation**: Pretrained backbone for downstream robotic fine-tuning (e.g., RoboTwin)
- **Research**: Studying the synergy between 3D spatial understanding and robot action learning

## Citation

```bibtex
@article{lin2026posevla,
  title={PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies},
  author={Lin, Haitao and Yu, Hanyang and Huang, Jingshun and Zhang, He and Ling, Yonggen and Tan, Ping and Xue, Xiangyang and Fu, Yanwei},
  journal={arXiv preprint arXiv:2602.19710},
  year={2026}
}
```

## License

This model is released under the [Apache 2.0 License](https://www.apache.org/licenses/LICENSE-2.0).