---
license: mit
library_name: pointmapvla
pipeline_tag: robotics
tags:
  - robotics
  - vision-language-action
  - pointmap
  - qwen3.5
  - flow-matching
  - robocasa
---

# PointmapVLA-RoboCasa

PointmapVLA fine-tuned on the 24 RoboCasa kitchen tasks (MimicGen-300 demonstrations).
This is the checkpoint behind the RoboCasa result of the paper. Code, recipes and the
evaluation harness: [PointmapVLA repository](https://github.com/DAVIAN-Robotics/PointmapVLA).

## What it is

PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language
backbone with a **pointmap encoder** (a copy of the vision tower fed raw metric XYZ
pointmaps in the robot-base frame, features added patch-wise to the RGB tokens) and a
**32-layer layer-wise flow-matching DiT action expert** (4 Euler steps at inference).
6.08 B parameters, bf16.

This checkpoint predicts chunks of 16 RoboCasa actions (15-D) from three camera views
(left, right, wrist), the instruction and an 18-D state, from which an 11-D end-effector
anchor is fed to the action expert.

## Provenance

* Initialised from [`DAVIAN-Robotics/PointmapVLA-Pretrained`](https://huggingface.co/DAVIAN-Robotics/PointmapVLA-Pretrained)
  (Stage 1 + Stage 2 pretraining on 76.3 M robot frames and 10.7 M 3D spatial VQA samples).
* Fine-tuned for 20,000 optimizer steps, global batch 128 (4 GPUs x 32), AdamW, cosine
  schedule with 1,000 warm-up steps, learning rates 1e-5 (backbone, pointmap encoder) and
  1e-4 (action expert, projections), DeepSpeed ZeRO-2, bf16.
* Data: [`DAVIAN-Robotics/robocasa-MG_300-r6`](https://huggingface.co/datasets/DAVIAN-Robotics/robocasa-MG_300-r6)
  + [`DAVIAN-Robotics/robocasa-MG_300-pointmap`](https://huggingface.co/datasets/DAVIAN-Robotics/robocasa-MG_300-pointmap)
  (7,200 episodes, 2.08 M frames, 3 cameras).
* On load from the pretrained checkpoint the four action-head tensors whose shape depends
  on the layout were remapped (shared end-effector/gripper slots copied, new slots fresh);
  all other weights transferred unchanged.
* Prompt: `Task: {instruction} Mode: <end_effector> Pointmap_origin: <robot>` with
  `enable_thinking: false` (an explicit empty `<think>` block).
* Exact config: `examples/simBenchmarks/Robocasa_kitchen/train_files/pointmapvla_robocasa_kitchen.yaml`
  in the repository (`_80gb.yaml` is the same recipe with gradient accumulation).

## Files

```
config.yaml                        architecture and the data recipe (release schema)
dataset_statistics.json            action/state mean/std/min/max/quantiles used for (un)normalisation
base_vlm/                          Qwen3.5-4B config, tokenizer and processor only; no weights
checkpoints/
  model-00001-of-00003.safetensors
  model-00002-of-00003.safetensors
  model-00003-of-00003.safetensors   ~11 GB in total, bf16
  model.safetensors.index.json
```

State-dict key groups: `qwen_vl_interface.model.*` (backbone), `pointmap_encoder.*`,
`action_model.*` (DiT expert), `project_layers.*` (per-layer projections). The language
model head is tied to the token embedding and stored once; the index records it under
`metadata.tied_weights` and the loader re-ties it.

## How to load

```python
from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA

model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-RoboCasa")
out = model.predict_action(examples, noise_seed=0)      # {"normalized_actions": (B, 16, 15)}
```

`examples[i]` is a dict with `image` (3 HWC uint8 arrays, 224 x 224), `lang`, `state`
(`(1, 18)` float32, normalised), `pointmap` (`(3, 256, 256, 3)` float16 metres,
robot-base frame) and `pointmap_valid` (`(3, 256, 256)` bool). Un-normalise with
`dataset_statistics.json`. In practice, run the policy server from the repository, which
does the un-normalisation and the r6 -> axis-angle conversion:

```bash
python deployment/model_server/server_policy.py --ckpt DAVIAN-Robotics/PointmapVLA-RoboCasa --port 20000 --use_bf16
```

## Evaluation

Protocol: 24 tasks x 50 episodes, max 720 steps, action chunk 16 executed in full, 4 Euler
steps, environment seed 42 (episode i uses `42*100000 + i`), three repetitions with policy
noise seeds 0, 1, 2; the number is the mean (± sd) over repetitions. One repetition alone
typically lands within about ±1.3 points of the mean.

```bash
CKPT=DAVIAN-Robotics/PointmapVLA-RoboCasa GPUS=0,1 scripts/eval/run_eval_robocasa.sh
```

| Benchmark | Paper | Our reproduction |
|---|---|---|
| RoboCasa kitchen-24, success rate (%) | 78.2 | 77.8 ± 0.6 (3 repetitions) |

## Intended use and limitations

* Simulation benchmark checkpoint for the RoboCasa kitchen-24 tasks (Panda-Omron, MG-300
  distribution). Not validated on physical robots.
* Layouts are fixed: action 15-D (EE position 3 | EE rotation r6 6 | gripper 1 | base
  motion 4 | control mode 1), horizon 16; state 18-D (base position 3 | base quaternion 4 |
  EE position 3 | EE rotation r6 6 | gripper 2). For another robot or layout, fine-tune
  `PointmapVLA-Pretrained` instead.
* Requires metric depth with known intrinsics and camera-to-world extrinsics for the
  pointmap input; the model was trained with pointmaps present.

## License

MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed
under the Apache License 2.0. See `NOTICE` in the repository.

## Citation

TBD. The paper is under double-blind review; the BibTeX entry and author list will be
added at camera-ready.