| --- |
| license: mit |
| library_name: pointmapvla |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - vision-language-action |
| - pointmap |
| - qwen3.5 |
| - flow-matching |
| --- |
| |
| # PointmapVLA-Pretrained |
|
|
| The pretrained PointmapVLA model: the initialisation for every fine-tuned PointmapVLA |
| checkpoint. Code, fine-tuning recipes and the evaluation harness are in the |
| [PointmapVLA repository](https://github.com/DAVIAN-Robotics/PointmapVLA). |
|
|
| ## What it is |
|
|
| PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language |
| backbone with two additions: |
|
|
| * a **pointmap encoder**, a copy of the vision tower that consumes raw metric XYZ pointmaps |
| in the robot-base frame (built from depth and camera parameters) and whose features are |
| added patch-wise to the RGB tokens, and |
| * a **32-layer layer-wise flow-matching DiT action expert** that reads the backbone's |
| hidden states at every layer through per-layer projections and predicts a chunk of |
| normalised actions (4 Euler steps at inference). |
|
|
| 6.08 B parameters, bf16. This checkpoint has a generic 32-D action head (17 real |
| dimensions: end-effector position 3, rotation as 6-D (first two rows of the rotation |
| matrix) 6, gripper 1, joints 7, then padding) and a 17-D state head. It is **not meant to |
| be run as is**: fine-tune it on your target robot and action layout. |
|
|
| ## Provenance |
|
|
| Two pretraining stages on 76.3 M pointmap-enriched robot frames and 10.7 M |
| pointmap-paired 3D spatial VQA samples, global batch 368: |
|
|
| 1. **Stage 1**, 50,000 steps: next-token prediction on the VQA data and discrete |
| (tokenised) action prediction on the robot data. |
| 2. **Stage 2**, 50,000 steps: flow-matching action prediction with the DiT expert, |
| co-trained with the VQA data. |
|
|
| The pretraining data and code are not part of this release. The configuration stored in |
| `config.yaml` records the architecture and the data recipe (prompt template, pointmap |
| options, 256 x 320 pointmaps); the benchmark fine-tuning configs override the head |
| dimensions and pointmap size. |
|
|
| ## Files |
|
|
| ``` |
| config.yaml architecture and data recipe (release schema) |
| dataset_statistics.json pretraining action/state statistics (informational) |
| base_vlm/ Qwen3.5-4B config, tokenizer and processor only; no weights |
| checkpoints/ |
| model-00001-of-00003.safetensors |
| model-00002-of-00003.safetensors |
| model-00003-of-00003.safetensors ~11 GB in total, bf16 |
| model.safetensors.index.json |
| ``` |
|
|
| State-dict key groups: `qwen_vl_interface.model.*` (backbone), `pointmap_encoder.*`, |
| `action_model.*` (DiT expert), `project_layers.*` (per-layer projections). The language |
| model head is tied to the token embedding and stored once; the index records it under |
| `metadata.tied_weights` and the loader re-ties it. |
|
|
| ## How to load |
|
|
| ```python |
| from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA |
| |
| model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-Pretrained") |
| ``` |
|
|
| `from_pretrained` also accepts a local directory or a weight file inside its |
| `checkpoints/`. For fine-tuning, set `trainer.pretrained_checkpoint` in a training config |
| to this repo id (the shipped configs already do) and run |
|
|
| ```bash |
| scripts/train.sh examples/simBenchmarks/Robocasa_kitchen/train_files/pointmapvla_robocasa_kitchen.yaml |
| scripts/train.sh examples/simBenchmarks/LIBERO/train_files/pointmapvla_libero.yaml |
| ``` |
|
|
| On load into a fine-tuning config, the four action-head tensors whose shape depends on |
| the action/state layout (state encoder input, action encoder input, action decoder output |
| weight and bias) are remapped: the slots shared by both layouts (end-effector position, |
| r6 rotation, gripper) are copied, the rest are freshly initialised. Everything else |
| transfers unchanged. Four GPUs is the minimum for fine-tuning; see `docs/training.md` in |
| the repository. |
|
|
| ## Evaluation |
|
|
| This checkpoint is not evaluated directly. The fine-tuned checkpoints derived from it: |
|
|
| | Fine-tune | Benchmark | Paper | Our reproduction | |
| |---|---|---|---| |
| | [`PointmapVLA-RoboCasa`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-RoboCasa) | RoboCasa kitchen-24 | 78.2 | 77.8 ± 0.6 | |
| | [`PointmapVLA-LIBERO`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-LIBERO) | LIBERO (4 suites) | 99.0 | 99.1 ± 0.1 | |
| | [`PointmapVLA-LIBERO`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-LIBERO) | LIBERO-Plus | 92.4 | 92.4 ± 0.3 | |
|
|
| ## Intended use and limitations |
|
|
| * Research on 3D-aware vision-language-action models; initialisation for fine-tuning. |
| * Inputs: RGB views, a language instruction, a proprioceptive state and metric pointmaps |
| in the robot-base frame (depth + intrinsics + camera-to-world extrinsics + base pose). |
| Without depth there is no pointmap; the model was trained with pointmaps present. |
| * The 32-D / 17-D head layout is a pretraining convenience. Fine-tune before use on any |
| robot; the released fine-tunes target simulation benchmarks only. |
| * This checkpoint is **not a servable policy**: the evaluation server needs the |
| fine-tuning data registry to un-normalise actions, and the pretraining embodiments |
| (`interndata_m1`, `droid` in `dataset_statistics.json`) are not part of this release. |
| Use it only as `trainer.pretrained_checkpoint`. |
| * Not validated on physical robots in this release. Actions are predicted in normalised |
| space and must be un-normalised with statistics of the fine-tuning data. |
|
|
| ## License |
|
|
| MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed |
| under the Apache License 2.0. See `NOTICE` in the repository. |
|
|
| ## Citation |
|
|
| TBD. The paper is under double-blind review; the BibTeX entry and author list will be |
| added at camera-ready. |
|
|
