| --- |
| license: mit |
| library_name: pointmapvla |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - vision-language-action |
| - pointmap |
| - qwen3.5 |
| - flow-matching |
| - libero |
| --- |
| |
| # PointmapVLA-LIBERO |
|
|
| PointmapVLA fine-tuned on the four LIBERO suites (spatial, object, goal, 10). This is the |
| checkpoint behind the LIBERO result of the paper and, evaluated zero-shot, the LIBERO-Plus |
| result. Code, recipes and the evaluation harness: |
| [PointmapVLA repository](https://github.com/DAVIAN-Robotics/PointmapVLA). |
|
|
| ## What it is |
|
|
| PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language |
| backbone with a **pointmap encoder** (a copy of the vision tower fed raw metric XYZ |
| pointmaps in the robot-base frame, features added patch-wise to the RGB tokens) and a |
| **32-layer layer-wise flow-matching DiT action expert** (4 Euler steps at inference). |
| 6.08 B parameters, bf16. |
|
|
| This checkpoint predicts chunks of 8 LIBERO actions (10-D) from two camera views |
| (agentview, wrist), the instruction and an 11-D end-effector state. |
|
|
| ## Provenance |
|
|
| * Initialised from [`DAVIAN-Robotics/PointmapVLA-Pretrained`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-Pretrained) |
| (Stage 1 + Stage 2 pretraining on 76.3 M robot frames and 10.7 M 3D spatial VQA samples). |
| * Fine-tuned for 30,000 optimizer steps, global batch 128 (4 GPUs x 32), AdamW, cosine |
| schedule with 1,500 warm-up steps, learning rates 1e-5 (backbone, pointmap encoder) and |
| 1e-4 (action expert, projections), DeepSpeed ZeRO-2, bf16. One policy for all four suites. |
| * Data: [`DAVIAN-Robotics/libero-r6`](https://huggingface.co/datasets/DAVIAN-Robotics/libero-r6) |
| + [`DAVIAN-Robotics/libero-r6-pointmap`](https://huggingface.co/datasets/DAVIAN-Robotics/libero-r6-pointmap) |
| (1,693 episodes over 40 tasks, 2 cameras). |
| * On load from the pretrained checkpoint the four action-head tensors whose shape depends |
| on the layout were remapped (shared end-effector/gripper slots copied, new slots fresh); |
| all other weights transferred unchanged. |
| * Prompt: `Task: {instruction} Mode: <end_effector> Pointmap_origin: <robot>`, chat |
| template default for thinking (no `enable_thinking` key; differs from the RoboCasa |
| checkpoint and is recorded in `config.yaml`). |
| * Exact config: `examples/simBenchmarks/LIBERO/train_files/pointmapvla_libero.yaml` in |
| the repository (`_80gb.yaml` is the same recipe with gradient accumulation). |
|
|
| ## Files |
|
|
| ``` |
| config.yaml architecture and the data recipe (release schema) |
| dataset_statistics.json action/state mean/std/min/max/quantiles used for (un)normalisation |
| base_vlm/ Qwen3.5-4B config, tokenizer and processor only; no weights |
| checkpoints/ |
| model-00001-of-00003.safetensors |
| model-00002-of-00003.safetensors |
| model-00003-of-00003.safetensors ~11 GB in total, bf16 |
| model.safetensors.index.json |
| ``` |
|
|
| State-dict key groups: `qwen_vl_interface.model.*` (backbone), `pointmap_encoder.*`, |
| `action_model.*` (DiT expert), `project_layers.*` (per-layer projections). The language |
| model head is tied to the token embedding and stored once; the index records it under |
| `metadata.tied_weights` and the loader re-ties it. |
|
|
| ## How to load |
|
|
| ```python |
| from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA |
| |
| model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-LIBERO") |
| out = model.predict_action(examples) # {"normalized_actions": (B, 8, 10)} |
| ``` |
|
|
| `examples[i]` is a dict with `image` (2 HWC uint8 arrays, 224 x 224, agentview and wrist, |
| rotated 180° as in the LIBERO datasets), `lang`, `state` (`(1, 11)` float32, normalised), |
| `pointmap` (`(2, 256, 256, 3)` float16 metres, robot-base frame) and `pointmap_valid` |
| (`(2, 256, 256)` bool). Un-normalise with `dataset_statistics.json`. In practice, run the |
| policy server from the repository, which does the un-normalisation and the r6 -> |
| axis-angle conversion: |
|
|
| ```bash |
| python deployment/model_server/server_policy.py --ckpt DAVIAN-Robotics/PointmapVLA-LIBERO --port 21000 --use_bf16 --seed 7 |
| ``` |
|
|
| ## Evaluation |
|
|
| Protocol: action chunk 8 executed in full, 4 Euler steps, K = 12 parallel environments, |
| server + environment seeds 7, 8, 9 (one run each); numbers are the mean (± sd) over the |
| three seeds. LIBERO: 4 suites x 10 tasks x 50 trials = 2,000 episodes. LIBERO-Plus |
| (zero-shot, same checkpoint): 10,030 perturbed tasks x 1 trial (2402 / 2518 / 2591 / 2519 |
| per suite). One seed alone typically lands within about ±0.2 points of the LIBERO mean. |
|
|
| ```bash |
| for s in 7 8 9; do CKPT=DAVIAN-Robotics/PointmapVLA-LIBERO BENCH=libero SEED=$s scripts/eval/run_eval_libero.sh; done |
| for s in 7 8 9; do CKPT=DAVIAN-Robotics/PointmapVLA-LIBERO BENCH=libero_plus SEED=$s GPUS=0,1 scripts/eval/run_eval_libero.sh; done |
| ``` |
|
|
| | Benchmark | Paper | Our reproduction | |
| |---|---|---| |
| | LIBERO (4 suites), success rate (%) | 99.0 | 99.1 ± 0.1 (seeds 7, 8, 9) | |
| | LIBERO-Plus, success rate (%) | 92.4 | 92.4 ± 0.3 (seeds 7, 8, 9) | |
|
|
| ## Intended use and limitations |
|
|
| * Simulation benchmark checkpoint for LIBERO and LIBERO-Plus (Franka Panda). Not |
| validated on physical robots. |
| * Layouts are fixed: action 10-D (EE position 3 | EE rotation r6 6 | gripper 1), horizon |
| 8; state 11-D (EE position 3 | EE rotation r6 6 | gripper 2). For another robot or |
| layout, fine-tune `PointmapVLA-Pretrained` instead. |
| * Requires metric depth with known intrinsics and camera-to-world extrinsics for the |
| pointmap input; the model was trained with pointmaps present. LIBERO images are |
| rotated by 180°, encoded in the datasets as a negative `fx`. |
|
|
| ## License |
|
|
| MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed |
| under the Apache License 2.0. See `NOTICE` in the repository. |
|
|
| ## Citation |
|
|
| TBD. The paper is under double-blind review; the BibTeX entry and author list will be |
| added at camera-ready. |
|
|
