File size: 3,642 Bytes
f4a5f2c | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 | ---
library_name: transformers
base_model: allenai/MolmoAct2
datasets:
- Jiafei1224/as
tags:
- molmoact2
- robotics
- image-text-to-text
- lerobot
- bimanual
---
# YAMBox2 groceries — MolmoAct2 regular
This is the regular MolmoAct2 checkpoint fine-tuned on
[`Jiafei1224/as`](https://huggingface.co/datasets/Jiafei1224/as) for a bimanual
YAM robot putting groceries into a container. This repository contains the
10,000-step checkpoint from the 50,000-step training run.
## Policy contract
- Base checkpoint: `allenai/MolmoAct2`
- Cameras, in order: `observation.images.top`, `observation.images.left`,
`observation.images.right`
- State: `observation.state`, 14 dimensions
- Output: 14-dimensional absolute joint targets
- Action horizon / returned steps: 30 / 30
- Normalization tag: `yam_dual_molmoact2`
- Grippers are not separately normalized
- Training sequence length: 784
- Depth reasoning: disabled
The converted action expert has a padded internal width of 32, but
`norm_stats.json` defines the real robot action width as 14 and `predict_action`
returns only those 14 dimensions.
## Install
```bash
pip install "torch" "transformers==5.14.1" accelerate pillow numpy safetensors
```
The model uses custom Transformers code included in this repository, so loading
requires `trust_remote_code=True`.
## Run one observation
```python
import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor
model_id = "hqfang/regular"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.bfloat16,
).to("cuda").eval()
# Keep this exact camera order.
images = [
Image.open("top.png").convert("RGB"),
Image.open("left.png").convert("RGB"),
Image.open("right.png").convert("RGB"),
]
state = np.asarray([0.0] * 14, dtype=np.float32) # replace with current joints
with torch.inference_mode():
result = model.predict_action(
processor=processor,
images=images,
task="put the groceries into the container",
state=state,
norm_tag="yam_dual_molmoact2",
expected_action_representation="absolute",
inference_action_mode="continuous",
num_steps=10,
n_action_steps=30,
)
actions = result.actions[0].float().cpu().numpy()
assert actions.shape == (30, 14)
```
`actions[t]` is an absolute 14-dimensional joint target, not a delta. Validate
joint ordering, limits, timing, and emergency-stop behavior before commanding
physical hardware.
## Open-loop check
The fixed-random open-loop evaluation used episode 2 from `Jiafei1224/as`
(1,488 frames, continuous inference, 10 flow steps). Raw action MSE was
`0.0012448359` at step 10,000. This is an offline reconstruction metric, not a
robot task-success measurement.
## Citation
```bibtex
@misc{fang2026molmoact2actionreasoningmodels,
title={MolmoAct2: Action Reasoning Models for Real-world Deployment},
author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna},
year={2026},
eprint={2605.02881},
archivePrefix={arXiv},
primaryClass={cs.RO}
}
```
|