Instructions to use twanghcmut/molmoact2-so101-left-arm-task3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use twanghcmut/molmoact2-so101-left-arm-task3 with LeRobot:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
MolmoAct2 SO101 Left Arm, 3-task
MolmoAct2-SO100_101 fully fine-tuned on
hungho77/so101-left-arm-task3 (three SO101
left-arm tasks merged into one set) with the LeRobot molmoact2 policy.
One checkpoint covers all three tasks; the language instruction selects the behaviour.
Same recipe as the 11-task twanghcmut/molmoact2-so101-left-arm-multitask,
same data and sample budget (768k) as the task3 checkpoints of pi0.5, GR00T N1.6 / N1.7 and X-VLA linked below.
License: this is a derivative of allenai/MolmoAct2-SO100_101; its terms apply.
Tasks
Use the exact wording, including trailing punctuation.
| # | Instruction | Episodes |
|---|---|---|
| 0 | Take the red cube out of the steel pot and place it into the orange hollow cube. |
51 |
| 1 | Use the left gripper to open the top drawer, pick up the red square block, place it inside the top drawer, and then close the drawer. |
57 |
| 2 | Open the drawer, pick up the red cube then stack on the blue cube |
51 |
159 episodes, 133,763 frames at 30 fps.
Robot
| State / action dim | 6: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll + gripper |
| Cameras | observation.images.top (360x640), observation.images.wrist (480x640); observation.images.top_depth is not used |
| Action | absolute joint targets, chunk of 30 (n_action_steps=30) |
| Prompt metadata | setup_type="single so100/so101 robotic arm in molmoact2", control_mode="absolute joint pose" |
| Normalization | this dataset's q01/q99 (QUANTILES) for state and action, in the pre/post-processor files; gripper normalized too |
Usage
from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy
from lerobot.policies.factory import make_pre_post_processors
from lerobot.configs.policies import PreTrainedConfig
repo = "twanghcmut/molmoact2-so101-left-arm-task3"
policy = MolmoAct2Policy.from_pretrained(repo).to("cuda").eval()
pre, post = make_pre_post_processors(PreTrainedConfig.from_pretrained(repo), pretrained_path=repo)
batch = {
"observation.images.top": top, # (1, 3, 360, 640) float in [0, 1]
"observation.images.wrist": wrist, # (1, 3, 480, 640) float in [0, 1]
"observation.state": state, # (1, 6) float32, raw joint units
"task": "Take the red cube out of the steel pot and place it into the orange hollow cube.",
}
actions = post(policy.predict_action_chunk(pre(batch))) # (1, 30, 6) absolute joint targets
VLA-RT serves MolmoAct2 as TensorRT engines, see VLA-RT.
Training
| Trainer | upstream huggingface/lerobot at commit b6ec0060, lerobot-train --policy.type=molmoact2 (launcher run_molmoact2_task3.sh) |
| Base | allenai/MolmoAct2-SO100_101, full fine-tune (train_mode_vlm=fft), embeddings frozen |
| Steps | 24,000 |
| Batch size | 32 (768k samples = 5.7 epochs) |
| LR | cosine with 200 warmup, peak 1e-5 (VLM) / 5e-5 (action expert), decays to 1e-6 |
| Optimizer | AdamW (0.9, 0.95), weight decay 0, per-group grad clip 1.0 |
| Flow matching | 8 flow timesteps per sample; knowledge insulation off |
| Augmentation | RandomRotation ±5°, ColorJitter |
| Precision | bfloat16, gradient checkpointing, torch.compile |
| Hardware | 1x H100 80GB, 55.6 GB peak, 23.7 h (about 3.6 s/step) |
| Final loss | 0.002-0.003 |
Loss at steps 2k / 4k / 8k / 12k / 16k / 20k / 24k: 0.015, 0.010, 0.007, 0.005, 0.003, 0.003, 0.003.
Open-loop evaluation (in-sample)
eval_so101_open_loop_lerobot.py (eval_molmoact2_task3.sh): two trajectories per task, walked in 16-step strides from
the dataset's own observations (images + observation.state), the first 16 predicted steps of each chunk compared with the
ground-truth actions, up to 400 frames per trajectory. MAE in the dataset's action units.
| traj | task | MAE | GR00T N1.6 | GR00T N1.7 | π0.5 |
|---|---|---|---|---|---|
| 0 | 0 red cube: pot -> hollow cube | 0.663 | 1.242 | 1.406 | 1.286 |
| 25 | 0 red cube: pot -> hollow cube | 0.790 | 1.476 | 1.542 | 1.690 |
| 51 | 1 drawer + red block | 5.133 | 1.211 | 1.359 | 1.239 |
| 80 | 1 drawer + red block | 3.441 | 1.531 | 1.601 | 1.704 |
| 108 | 2 drawer, stack red on blue | 0.545 | 1.262 | 1.245 | 1.345 |
| 133 | 2 drawer, stack red on blue | 0.689 | 1.347 | 1.486 | 1.488 |
| average | 1.877 (MSE 41.3) | 1.345 | 1.440 | 1.459 |
Divided by the absolute action std (29.97) the average is 6.26%.
Task 1 is the weak spot. On tasks 0 and 2 MolmoAct2 has the lowest error of all the task3 checkpoints (0.54-0.79, about half of GR00T / π0.5), but on both task-1 trajectories (the long drawer task, about 1,200 frames per episode) the error is 3-4x higher than the other models and dominates the average; the MSE (181.9 and 59.2) points at large excursions rather than a uniform offset. The two evaluated episodes are identical in the v3.0 tree used for training and in the v2.1 tree (same frames, same task index), so this is not a data mapping error. It has not been investigated further. Check task 1 before relying on this checkpoint for it.
Other task3 checkpoints: GR00T N1.6, GR00T N1.7, π0.5, X-VLA (its proprio is the previous action, so its MAE is not directly comparable). Each model was scored by its own framework's open-loop script following the same procedure.
Limitations
No held-out split: all 159 episodes were used for training, so these numbers measure fit, not generalization, and are not a success rate. Two trajectories per task is a small sample. This checkpoint has not been run on the real robot.
Deployment note
The data is 30 fps; a client interpolating between model steps should use smooth_step = control_hz / 30.
- Downloads last month
- 18
Model tree for twanghcmut/molmoact2-so101-left-arm-task3
Base model
allenai/MolmoAct2-SO100_101