Upload README.md with huggingface_hub

2bf05b6 verified 5 days ago

3.11 kB

license: apache-2.0
language:
  - en
tags:
  - robotics
  - 3d-detection
  - vision-language-action
  - pose-estimation
  - grounding
library_name: transformers
pipeline_tag: robotics

PoseVLA Stage-1: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies

Model Description

PoseVLA is a Vision-Language-Action (VLA) model that leverages universal 3D pose pretraining for generalizable robotic manipulation. This checkpoint is the Stage-1 pretrained model, jointly trained on large-scale 3D detection and robot action data.

Architecture: PaliGemma-3B + Action Expert (π0-based, trained from scratch) with Flow Matching
Pretraining Data: Omni3D, Omni6D, BOP, GraspClutter6D (3D tasks) + Agibot, InternData-A1 (robot actions)
Input: Multi-view RGB images + Depth priors + Camera intrinsics
Output: 3D object detection (Next-Token Prediction) / Robot actions (Flow Matching)

Usage

3D Grounding Inference

from posevla.modeling_posevla import PoseVLAPolicy, PoseVLAConfig, bin_tokenizer
from utils.mapping_token import decode_text_to_scene_with_tokenizer

# Load model
policy = PoseVLAPolicy.from_pretrained("hetolin/PoseVLA-stage1", local_files_only=False, config=posevla_config)
policy = policy.eval().to(torch.bfloat16).cuda()

# Inference
output_res = policy.forward_evaluate_ntp(batch)
pred_text = output_res["pred"][0]
pred_res = decode_text_to_scene_with_tokenizer(pred_text, bin_tokenizer)

Full Inference Script

See infer_grounding3d.py for complete real-world RGB-D inference pipeline.

Training Details

Hyperparameter	Value
Base model	PaliGemma-3B-pt-224
Action Expert	π0 (Flow Matching, from scratch)
Image resolution	224 × 224
Optimizer	AdamW
Learning rate	5e-5
Weight decay	1e-10
Precision	bf16
GPUs	16 × H20
Batch size	7 per GPU
Training steps	100K

Intended Use

3D Object Grounding: Open-vocabulary 3D detection from RGB-D images
Robot Manipulation: Pretrained backbone for downstream robotic fine-tuning (e.g., RoboTwin)
Research: Studying the synergy between 3D spatial understanding and robot action learning

Citation

@article{lin2026posevla,
  title={PoseVLA: Universal Pose Pretraining for Generalizable Vision-Language-Action Policies},
  author={Lin, Haitao and Yu, Hanyang and Huang, Jingshun and Zhang, He and Ling, Yonggen and Tan, Ping and Xue, Xiangyang and Fu, Yanwei},
  journal={arXiv preprint arXiv:2602.19710},
  year={2026}
}

License

This model is released under the Apache 2.0 License.