MolmoAct2 SO101 Left Arm, 3-task

MolmoAct2-SO100_101 fully fine-tuned on hungho77/so101-left-arm-task3 (three SO101 left-arm tasks merged into one set) with the LeRobot molmoact2 policy. One checkpoint covers all three tasks; the language instruction selects the behaviour.

Same recipe as the 11-task twanghcmut/molmoact2-so101-left-arm-multitask, same data and sample budget (768k) as the task3 checkpoints of pi0.5, GR00T N1.6 / N1.7 and X-VLA linked below.

License: this is a derivative of allenai/MolmoAct2-SO100_101; its terms apply.

Tasks

Use the exact wording, including trailing punctuation.

# Instruction Episodes
0 Take the red cube out of the steel pot and place it into the orange hollow cube. 51
1 Use the left gripper to open the top drawer, pick up the red square block, place it inside the top drawer, and then close the drawer. 57
2 Open the drawer, pick up the red cube then stack on the blue cube 51

159 episodes, 133,763 frames at 30 fps.

Robot

State / action dim 6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper
Cameras observation.images.top (360x640), observation.images.wrist (480x640); observation.images.top_depth is not used
Action absolute joint targets, chunk of 30 (n_action_steps=30)
Prompt metadata setup_type="single so100/so101 robotic arm in molmoact2", control_mode="absolute joint pose"
Normalization this dataset's q01/q99 (QUANTILES) for state and action, in the pre/post-processor files; gripper normalized too

Usage

from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.configs.policies import PreTrainedConfig

repo = "twanghcmut/molmoact2-so101-left-arm-task3"
policy = MolmoAct2Policy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(PreTrainedConfig.from_pretrained(repo), pretrained_path=repo)

batch = {
    "observation.images.top":   top,     # (1, 3, 360, 640) float in [0, 1]
    "observation.images.wrist": wrist,   # (1, 3, 480, 640) float in [0, 1]
    "observation.state":        state,   # (1, 6) float32, raw joint units
    "task": "Take the red cube out of the steel pot and place it into the orange hollow cube.",
}
actions = post(policy.predict_action_chunk(pre(batch)))  # (1, 30, 6) absolute joint targets

VLA-RT serves MolmoAct2 as TensorRT engines, see VLA-RT.

Training

Trainer upstream huggingface/lerobot at commit b6ec0060, lerobot-train --policy.type=molmoact2 (launcher run_molmoact2_task3.sh)
Base allenai/MolmoAct2-SO100_101, full fine-tune (train_mode_vlm=fft), embeddings frozen
Steps 24,000
Batch size 32 (768k samples = 5.7 epochs)
LR cosine with 200 warmup, peak 1e-5 (VLM) / 5e-5 (action expert), decays to 1e-6
Optimizer AdamW (0.9, 0.95), weight decay 0, per-group grad clip 1.0
Flow matching 8 flow timesteps per sample; knowledge insulation off
Augmentation RandomRotation ±5°, ColorJitter
Precision bfloat16, gradient checkpointing, torch.compile
Hardware 1x H100 80GB, 55.6 GB peak, 23.7 h (about 3.6 s/step)
Final loss 0.002-0.003

Loss at steps 2k / 4k / 8k / 12k / 16k / 20k / 24k: 0.015, 0.010, 0.007, 0.005, 0.003, 0.003, 0.003.

Open-loop evaluation (in-sample)

eval_so101_open_loop_lerobot.py (eval_molmoact2_task3.sh): two trajectories per task, walked in 16-step strides from the dataset's own observations (images + observation.state), the first 16 predicted steps of each chunk compared with the ground-truth actions, up to 400 frames per trajectory. MAE in the dataset's action units.

traj task MAE GR00T N1.6 GR00T N1.7 π0.5
0 0 red cube: pot -> hollow cube 0.663 1.242 1.406 1.286
25 0 red cube: pot -> hollow cube 0.790 1.476 1.542 1.690
51 1 drawer + red block 5.133 1.211 1.359 1.239
80 1 drawer + red block 3.441 1.531 1.601 1.704
108 2 drawer, stack red on blue 0.545 1.262 1.245 1.345
133 2 drawer, stack red on blue 0.689 1.347 1.486 1.488
average 1.877 (MSE 41.3) 1.345 1.440 1.459

Divided by the absolute action std (29.97) the average is 6.26%.

Task 1 is the weak spot. On tasks 0 and 2 MolmoAct2 has the lowest error of all the task3 checkpoints (0.54-0.79, about half of GR00T / π0.5), but on both task-1 trajectories (the long drawer task, about 1,200 frames per episode) the error is 3-4x higher than the other models and dominates the average; the MSE (181.9 and 59.2) points at large excursions rather than a uniform offset. The two evaluated episodes are identical in the v3.0 tree used for training and in the v2.1 tree (same frames, same task index), so this is not a data mapping error. It has not been investigated further. Check task 1 before relying on this checkpoint for it.

Other task3 checkpoints: GR00T N1.6, GR00T N1.7, π0.5, X-VLA (its proprio is the previous action, so its MAE is not directly comparable). Each model was scored by its own framework's open-loop script following the same procedure.

Limitations

No held-out split: all 159 episodes were used for training, so these numbers measure fit, not generalization, and are not a success rate. Two trajectories per task is a small sample. This checkpoint has not been run on the real robot.

Deployment note

The data is 30 fps; a client interpolating between model steps should use smooth_step = control_hz / 30.

Downloads last month
18
Safetensors
Model size
5B params
Tensor type
F32
·
BF16
·
Video Preview
loading

Model tree for twanghcmut/molmoact2-so101-left-arm-task3

Finetuned
(25)
this model