README.md

metadata

library_name: transformers
base_model: allenai/MolmoAct2
datasets:
  - Jiafei1224/as
tags:
  - molmoact2
  - robotics
  - image-text-to-text
  - lerobot
  - bimanual

YAMBox2 groceries — MolmoAct2 regular

This is the regular MolmoAct2 checkpoint fine-tuned on Jiafei1224/as for a bimanual YAM robot putting groceries into a container. This repository contains the 10,000-step checkpoint from the 50,000-step training run.

Policy contract

• Base checkpoint: allenai/MolmoAct2
• Cameras, in order: observation.images.top, observation.images.left, observation.images.right
• State: observation.state, 14 dimensions
• Output: 14-dimensional absolute joint targets
• Action horizon / returned steps: 30 / 30
• Normalization tag: yam_dual_molmoact2
• Grippers are not separately normalized
• Training sequence length: 784
• Depth reasoning: disabled

The converted action expert has a padded internal width of 32, but norm_stats.json defines the real robot action width as 14 and predict_action returns only those 14 dimensions.

Install

pip install "torch" "transformers==5.14.1" accelerate pillow numpy safetensors

The model uses custom Transformers code included in this repository, so loading requires trust_remote_code=True.

Run one observation

import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "hqfang/regular"

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    model_id,
    trust_remote_code=True,
    torch_dtype=torch.bfloat16,
).to("cuda").eval()

# Keep this exact camera order.
images = [
    Image.open("top.png").convert("RGB"),
    Image.open("left.png").convert("RGB"),
    Image.open("right.png").convert("RGB"),
]
state = np.asarray([0.0] * 14, dtype=np.float32)  # replace with current joints

with torch.inference_mode():
    result = model.predict_action(
        processor=processor,
        images=images,
        task="put the groceries into the container",
        state=state,
        norm_tag="yam_dual_molmoact2",
        expected_action_representation="absolute",
        inference_action_mode="continuous",
        num_steps=10,
        n_action_steps=30,
    )

actions = result.actions[0].float().cpu().numpy()
assert actions.shape == (30, 14)

actions[t] is an absolute 14-dimensional joint target, not a delta. Validate joint ordering, limits, timing, and emergency-stop behavior before commanding physical hardware.

Open-loop check

The fixed-random open-loop evaluation used episode 2 from Jiafei1224/as (1,488 frames, continuous inference, 10 flow steps). Raw action MSE was 0.0012448359 at step 10,000. This is an offline reconstruction metric, not a robot task-success measurement.

Citation

@misc{fang2026molmoact2actionreasoningmodels,
  title={MolmoAct2: Action Reasoning Models for Real-world Deployment},
  author={Haoquan Fang and Jiafei Duan and Donovan Clay and Sam Wang and Shuo Liu and Weikai Huang and Xiang Fan and Wei-Chuan Tsai and Shirui Chen and Yi Ru Wang and Shanli Xing and Jaemin Cho and Jae Sung Park and Ainaz Eftekhar and Peter Sushko and Karen Farley and Angad Wadhwa and Cole Harrison and Winson Han and Ying-Chun Lee and Eli VanderBilt and Rose Hendrix and Suveen Ellawela and Lucas Ngoo and Joyce Chai and Zhongzheng Ren and Ali Farhadi and Dieter Fox and Ranjay Krishna},
  year={2026},
  eprint={2605.02881},
  archivePrefix={arXiv},
  primaryClass={cs.RO}
}