Model Card for molmoact2

MolmoAct2 is an open robotics foundation model from the Allen Institute for AI (Ai2) that maps camera images and language instructions to robot action chunks. The LeRobot implementation supports training and evaluation of the regular MolmoAct2 model.

This policy has been trained and pushed to the Hub using LeRobot.

Learn how to train and run it in the LeRobot molmoact2 guide, or browse the full documentation.

Model Details

License: apache-2.0
Robot type: so_follower
Cameras: above, left, middle, right, wrist

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
`observation.images.middle`	VISUAL	`(3, 480, 640)`
`observation.images.wrist`	VISUAL	`(3, 480, 640)`
`observation.state`	STATE	`(6,)`

Outputs

Feature	Type	Shape
`action`	ACTION	`(6,)`

Training Dataset

Repository: justintiensmith/VLA_Reasoning_Training_Dataset_1200
Episodes: 1200
Frames: 612733
Frame rate: 30 FPS
Task(s): "Move the closed white cup into the bowl.", "Place the open white cup in the bowl.", "Put the open white cup in the bowl.", "Place a white cup in the bowl.", "Put a white cup in the bowl.", "Place the closed white cup in the bowl.", "Move the open white cup into the bowl.", "Move a white cup into the bowl.", "Put the closed white cup in the bowl.", "Put the upright white cup in the bowl.", "Move the upright white cup into the bowl.", "Place the upright white cup in the bowl.", "Put a yellow-white striped cup in the bowl.", "Move the closed yellow-white striped cup into the bowl.", "Place a yellow-white striped cup in the bowl.", "Move a yellow-white striped cup into the bowl.", "Place the closed yellow-white striped cup in the bowl.", "Put the closed yellow-white striped cup in the bowl.", "Move the upright yellow-white striped cup into the bowl.", "Place the upright yellow-white striped cup in the bowl.", "Move the upside-down yellow-white striped cup into the bowl.", "Put the upright yellow-white striped cup in the bowl.", "Put the upside-down yellow-white striped cup in the bowl.", "Place the upside-down yellow-white striped cup in the bowl.", "Place a brown cup in the bowl.", "Put the open brown cup in the bowl.", "Move a brown cup into the bowl.", "Place the open brown cup in the bowl.", "Put a brown cup in the bowl.", "Move the open brown cup into the bowl.", "Put the upside-down brown cup in the bowl.", "Place the upside-down brown cup in the bowl.", "Put the upright brown cup in the bowl.", "Place the upright brown cup in the bowl.", "Move the upside-down brown cup into the bowl.", "Move the upright brown cup into the bowl.", "Move the open glass spice container into the bowl.", "Place the closed glass spice container in the bowl.", "Put the closed glass spice container in the bowl.", "Move a glass spice container into the bowl.", "Place the open glass spice container in the bowl.", "Place a glass spice container in the bowl.", "Move the closed glass spice container into the bowl.", "Put a glass spice container in the bowl.", "Put the open glass spice container in the bowl.", "Put the upside-down glass spice container in the bowl.", "Move the upside-down glass spice container into the bowl.", "Place the upside-down glass spice container in the bowl.", "Move the green pen near the red block.", "Place the green pen next to the red block.", "Put the green pen beside the red block.", "Move the green pen to the front of the pink marker.", "Put the green pen on the table in front of the pink marker.", "Place the green pen in front of the pink marker.", "Move the green pen to the front of the blue block.", "Put the green pen on the table in front of the blue block.", "Place the green pen in front of the blue block.", "Move the green pen to the back of the pink marker.", "Place the green pen behind the pink marker.", "Move the green pen to the back of the blue block.", "Put the green pen on the table behind the blue block.", "Place the green pen to the left of the pink marker.", "Place the green pen to the left of the blue block.", "Place the green pen to the right of the pink marker.", "Place the green pen to the right of the blue block.", "Move the red block to the front of the pink marker.", "Put the red block on the table in front of the pink marker.", "Place the red block in front of the pink marker.", "Move the red block to the front of the blue block.", "Put the red block on the table in front of the blue block.", "Place the red block in front of the blue block.", "Place the red block next to the green pen.", "Put the red block beside the green pen.", "Move the red block near the green pen.", "Place the red block behind the pink marker.", "Move the red block to the back of the pink marker.", "Put the red block on the table behind the pink marker.", "Place the red block behind the blue block.", "Move the red block to the back of the blue block.", "Put the red block on the table behind the blue block.", "Place the red block to the left of the pink marker.", "Place the red block to the left of the blue block.", "Place the red block to the right of the pink marker.", "Place the red block to the right of the blue block.", "Move the pink marker to the front of the blue block.", "Put the pink marker on the table in front of the blue block.", "Put the pink marker on the table in front of the green pen.", "Move the pink marker to the front of the green pen.", "Place the pink marker in front of the green pen.", "Move the pink marker to the front of the red block.", "Place the pink marker in front of the red block.", "Move the pink marker to the back of the blue block.", "Put the pink marker on the table behind the blue block.", "Move the pink marker to the back of the green pen.", "Put the pink marker on the table behind the green pen.", "Put the pink marker on the table behind the red block.", "Place the pink marker behind the red block.", "Place the pink marker to the left of the blue block.", "Place the pink marker to the left of the green pen.", "Place the pink marker to the left of the red block.", "Place the pink marker to the right of the blue block.", "Place the pink marker to the right of the green pen.", "Place the pink marker to the right of the red block.", "Put the blue block on the table in front of the green pen.", "Move the blue block to the front of the green pen.", "Place the blue block in front of the green pen.", "Place the blue block in front of the red block.", "Put the blue block on the table in front of the red block.", "Move the blue block to the front of the red block.", "Put the blue block on the table in front of the pink marker.", "Place the blue block in front of the pink marker.", "Move the blue block to the front of the pink marker.", "Put the blue block on the table behind the green pen.", "Place the blue block behind the green pen.", "Move the blue block to the back of the green pen.", "Place the blue block behind the red block.", "Put the blue block on the table behind the red block.", "Move the blue block to the back of the pink marker.", "Put the blue block on the table behind the pink marker.", "Place the blue block to the left of the green pen.", "Place the blue block to the left of the red block.", "Place the blue block to the left of the pink marker.", "Place the blue block to the right of the green pen.", "Place the blue block to the right of the red block.", "Place the blue block to the right of the pink marker.", "Put the crushed Coke can in the bowl.", "Pick up the crushed Pepsi can and place it in the bowl.", "Move the crushed Coke can into the bowl.", "Put the crushed Pepsi can in the bowl.", "Pick up the crushed Coke can and place it in the bowl.", "Move the crushed Pepsi can into the bowl.", "Pick up the crushed soda can furthest from the green pen and place it in the bowl.", "Pick up the crushed soda can furthest from the yellow pen and place it in the bowl.", "Pick up the crushed soda can furthest from the airplane toy with propellers and place it in the bowl.", "Pick up the crushed soda can furthest from the airplane toy without propellers and place it in the bowl.", "Pick up the crushed soda can furthest from the green pencil sharpener and place it in the bowl.", "Pick up the crushed soda can furthest from the red pencil sharpener and place it in the bowl.", "Move the crushed soda can lying between the green pencil sharpener and the green pen into the bowl.", "Pick up the crushed soda can positioned between the airplane toy with propellers and the green pencil sharpener, then place it in the bowl.", "Pick up the crushed soda can positioned between the airplane toy with propellers and the green pen, then place it in the bowl.", "Move the crushed soda can lying between the green pencil sharpener and the yellow pen into the bowl.", "Move the crushed soda can lying between the red pencil sharpener and the airplane toy with propellers into the bowl.", "Pick up the crushed soda can positioned between the airplane toy without propellers and the yellow pen, then place it in the bowl.", "Pick up the crushed soda can positioned between the green pen and the red pencil sharpener, then place it in the bowl.", "Put the crushed soda can between the red pencil sharpener and the airplane toy without propellers in the bowl.", "Put the crushed soda can between the green pen and the airplane toy without propellers in the bowl.", "Move the crushed soda can lying between the yellow pen and the red pencil sharpener into the bowl.", "Put the crushed soda can between the airplane toy without propellers and the green pencil sharpener in the bowl.", "Move the crushed soda can lying between the yellow pen and the airplane toy with propellers into the bowl.", "Pick up the crushed soda can positioned between the green pencil sharpener and the green pen, then place it in the bowl.", "Move the crushed soda can lying between the airplane toy with propellers and the green pen into the bowl.", "Pick up the crushed soda can positioned between the red pencil sharpener and the airplane toy with propellers, then place it in the bowl.", "Move the crushed soda can lying between the airplane toy without propellers and the yellow pen into the bowl.", "Put the crushed soda can between the green pen and the red pencil sharpener in the bowl.", "Pick up the crushed soda can positioned between the red pencil sharpener and the airplane toy without propellers, then place it in the bowl.", "Pick up the pencil sharpener nearest to the crushed Coke can and place it in the bowl.", "Move the pencil sharpener that is closest to the crushed Pepsi can into the bowl.", "Move the pencil sharpener that is closest to the green pen into the bowl.", "Put the pencil sharpener closest to the yellow pen in the bowl.", "Put the pencil sharpener closest to the airplane toy with propellers in the bowl.", "Move the pencil sharpener that is closest to the airplane toy without propellers into the bowl.", "Move the pencil sharpener that is closest to the crushed Coke can into the bowl.", "Put the pencil sharpener closest to the crushed Pepsi can in the bowl.", "Pick up the pencil sharpener nearest to the airplane toy with propellers and place it in the bowl.", "Pick up the pencil sharpener nearest to the airplane toy without propellers and place it in the bowl.", "Put the pencil sharpener closest to the crushed Coke can in the bowl.", "Pick up the pencil sharpener nearest to the crushed Pepsi can and place it in the bowl.", "Pick up the pencil sharpener furthest from the green pen and place it in the bowl.", "Pick up the pencil sharpener furthest from the yellow pen and place it in the bowl.", "Pick up the pencil sharpener furthest from the airplane toy with propellers and place it in the bowl.", "Pick up the pencil sharpener furthest from the airplane toy without propellers and place it in the bowl.", "Pick up the pencil sharpener furthest from the crushed Coke can and place it in the bowl.", "Pick up the pencil sharpener furthest from the crushed Pepsi can and place it in the bowl.", "Move the red pencil sharpener into the bowl.", "Put the red pencil sharpener in the bowl.", "Pick up the green pencil sharpener and place it in the bowl.", "Move the green pencil sharpener into the bowl.", "Put the green pencil sharpener in the bowl.", "Pick up the red pencil sharpener and place it in the bowl.", "Pick up the pen nearest to the green pencil sharpener and place it in the bowl.", "Put the pen closest to the red pencil sharpener in the bowl.", "Pick up the pen nearest to the crushed Coke can and place it in the bowl.", "Put the pen closest to the crushed Pepsi can in the bowl.", "Pick up the pen nearest to the airplane toy with propellers and place it in the bowl.", "Move the pen that is closest to the airplane toy without propellers into the bowl.", "Move the pen that is closest to the green pencil sharpener into the bowl.", "Move the pen that is closest to the red pencil sharpener into the bowl.", "Put the pen closest to the crushed Coke can in the bowl.", "Move the pen that is closest to the crushed Pepsi can into the bowl.", "Put the pen closest to the airplane toy with propellers in the bowl.", "Put the pen closest to the airplane toy without propellers in the bowl.", "Pick up the pen furthest from the crushed Coke can and place it in the bowl.", "Pick up the pen furthest from the crushed Pepsi can and place it in the bowl.", "Pick up the pen furthest from the airplane toy with propellers and place it in the bowl.", "Pick up the pen furthest from the airplane toy without propellers and place it in the bowl.", "Pick up the pen furthest from the green pencil sharpener and place it in the bowl.", "Pick up the pen furthest from the red pencil sharpener and place it in the bowl.", "Pick up the pen positioned between the airplane toy without propellers and the green pencil sharpener, then place it in the bowl.", "Move the pen lying between the crushed Pepsi can and the airplane toy with propellers into the bowl.", "Put the pen between the green pencil sharpener and the crushed Coke can in the bowl.", "Pick up the pen positioned between the airplane toy with propellers and the green pencil sharpener, then place it in the bowl.", "Pick up the pen positioned between the airplane toy with propellers and the crushed Coke can, then place it in the bowl.", "Put the pen between the green pencil sharpener and the crushed Pepsi can in the bowl.", "Move the pen lying between the red pencil sharpener and the airplane toy with propellers into the bowl.", "Put the pen between the airplane toy without propellers and the crushed Pepsi can in the bowl.", "Move the pen lying between the crushed Coke can and the red pencil sharpener into the bowl.", "Move the pen lying between the red pencil sharpener and the airplane toy without propellers into the bowl.", "Move the pen lying between the green pencil sharpener and the crushed Coke can into the bowl.", "Put the pen between the airplane toy with propellers and the green pencil sharpener in the bowl.", "Move the pen lying between the airplane toy with propellers and the crushed Coke can into the bowl.", "Pick up the pen positioned between the green pencil sharpener and the crushed Pepsi can, then place it in the bowl.", "Pick up the pen positioned between the red pencil sharpener and the airplane toy with propellers, then place it in the bowl.", "Move the pen lying between the airplane toy without propellers and the crushed Pepsi can into the bowl.", "Pick up the pen positioned between the red pencil sharpener and the airplane toy without propellers, then place it in the bowl.", "Put the pen between the crushed Coke can and the airplane toy without propellers in the bowl.", "Put the pen between the crushed Pepsi can and the red pencil sharpener in the bowl.", "Put the airplane toy closest to the green pen in the bowl.", "Pick up the airplane toy nearest to the yellow pen and place it in the bowl.", "Put the airplane toy closest to the green pencil sharpener in the bowl.", "Put the airplane toy closest to the red pencil sharpener in the bowl.", "Move the airplane toy that is closest to the crushed Coke can into the bowl.", "Move the airplane toy that is closest to the crushed Pepsi can into the bowl.", "Move the airplane toy that is closest to the green pen into the bowl.", "Move the airplane toy that is closest to the green pencil sharpener into the bowl.", "Pick up the airplane toy nearest to the crushed Pepsi can and place it in the bowl.", "Move the airplane toy that is closest to the yellow pen into the bowl.", "Pick up the airplane toy without propellers and place it in the bowl.", "Put the airplane toy with propellers in the bowl.", "Put the airplane toy without propellers in the bowl.", "Move the airplane toy with propellers into the bowl.", "Move the airplane toy without propellers into the bowl.", "Pick up the airplane toy with propellers and place it in the bowl.", "Pick up the airplane toy positioned between the green pen and the red pencil sharpener, then place it in the bowl.", "Pick up the airplane toy positioned between the yellow pen and the crushed Coke can, then place it in the bowl.", "Move the airplane toy lying between the crushed Pepsi can and the red pencil sharpener into the bowl.", "Pick up the airplane toy positioned between the green pencil sharpener and the yellow pen, then place it in the bowl.", "Pick up the airplane toy positioned between the yellow pen and the crushed Pepsi can, then place it in the bowl.", "Move the airplane toy lying between the green pen and the green pencil sharpener into the bowl.", "Put the airplane toy between the crushed Coke can and the green pen in the bowl.", "Move the airplane toy lying between the crushed Coke can and the green pencil sharpener into the bowl.", "Move the airplane toy lying between the green pen and the red pencil sharpener into the bowl.", "Put the airplane toy between the crushed Pepsi can and the red pencil sharpener in the bowl.", "Put the airplane toy between the green pencil sharpener and the yellow pen in the bowl.", "Put the airplane toy between the green pencil sharpener and the crushed Pepsi can in the bowl.", "Move the airplane toy lying between the red pencil sharpener and the yellow pen into the bowl.", "Move the airplane toy lying between the crushed Pepsi can and the green pen into the bowl.", "Pick up the airplane toy positioned between the red pencil sharpener and the crushed Coke can, then place it in the bowl.", "Pick up the airplane toy positioned between the green pen and the green pencil sharpener, then place it in the bowl.", "Put the third object from the right in the bowl.", "Pick up the object that is third from the right and place it in the bowl.", "Move the first item counted from the left into the bowl.", "Pick up the object that is second from the left and place it in the bowl.", "Put the second object from the left in the bowl.", "Move the second item counted from the left into the bowl.", "Put the object that is second from the right in the bowl.", "Pick up the object that is third from the left and place it in the bowl.", "Move the second from the right object into the bowl.", "Put the object that is third from the left in the bowl.", "Pick up the object that is second from the right and place it in the bowl.", "Move the third from the left object into the bowl.", "Pick up the object that is fourth from the left and place it in the bowl.", "Move the first from the right object into the bowl.", "Put the object that is fourth from the left in the bowl.", "Pick up the object that is first from the right and place it in the bowl.", "Move the fourth from the left object into the bowl.", "Put the object that is first from the right in the bowl.", "Move the fourth item counted from the right into the bowl.", "Pick up the object that is first from the left and place it in the bowl.", "Put the fourth object from the right in the bowl.", "Move the blue tape ball into the bowl.", "Put the blue tape ball in the bowl.", "Pick up the blue tape ball and place it in the bowl.", "Put the second object from the right in the bowl.", "Put the first object from the left in the bowl.", "Pick up the object that is fourth from the right and place it in the bowl.", "Move the third item counted from the right into the bowl.", "Move the green highlighter into the bowl.", "Put the green highlighter in the bowl.", "Pick up the green highlighter and place it in the bowl.", "Put the yellow pen in the bowl.", "Pick up the yellow pen and place it in the bowl.", "Move the yellow pen into the bowl.", "Move the second item counted from the right into the bowl.", "Leave exactly one object in the bowl.", "Make the bowl contain exactly two objects.", "Adjust the bowl so that it contains two objects.", "Pick up the blue pen and place it in the bowl.", "Move the blue pen into the bowl.", "Put the blue pen in the bowl.", "Make the bowl contain exactly zero objects.", "Adjust the bowl so that it contains zero objects.", "Leave no objects in the bowl.", "Remove the blue pen from the bowl.", "Take the blue pen out of the bowl.", "Move the blue pen from the bowl to the table.", "Leave exactly two objects in the bowl.", "Leave exactly three objects in the bowl.", "Put the yellow block in the bowl.", "Pick up the yellow block and place it in the bowl.", "Move the yellow block into the bowl.", "Remove the yellow block from the bowl.", "Move the yellow block from the bowl to the table.", "Take the yellow block out of the bowl.", "Remove the green pencil sharpener from the bowl.", "Move the green pencil sharpener from the bowl to the table.", "Take the green pencil sharpener out of the bowl.", "Remove the airplane toy without propellers from the bowl.", "Take the airplane toy without propellers out of the bowl.", "Move the airplane toy without propellers from the bowl to the table.", "Move the object that is smallest into the bowl.", "Pick up the smallest item and place it in the bowl.", "Put the smallest object in the bowl.", "Pick up the green block and place it in the bowl.", "Move the green block into the bowl.", "Put the green block in the bowl.", "Put the juggling ball in the bowl.", "Move the juggling ball into the bowl.", "Pick up the juggling ball and place it in the bowl.", "Put the largest object in the bowl.", "Pick up the largest item and place it in the bowl.", "Move the object that is largest into the bowl."

Training Configuration

Setting	Value
Training steps	102128
Batch size	24
Optimizer	adamw
Learning rate	1e-05
Seed	1000
LeRobot version	0.6.0

How to Get Started with the Model

New to LeRobot? These guides cover the full workflow:

Install LeRobot — set up the lerobot package.
Hardware setup — assemble, wire, and calibrate your robot and cameras.
Record data & train a policy — the end-to-end imitation-learning walkthrough.
CLI cheat-sheet — quick reference for the lerobot-* commands.

The short version to run and train this policy:

Run the policy on your robot

lerobot-rollout \
  --strategy.type=base \
  --robot.type=so_follower \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ <camera_1>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}, <camera_2>: {type: opencv, index_or_path: <index_or_path>, width: 640, height: 480, fps: 30}}" \
  --policy.path=justintiensmith/molmoact2_VLA_Reasoning_Training_Dataset_1200 \
  --task="Move the closed white cup into the bowl." \
  --duration=60

Replace the remaining <...> placeholders with your own values: --robot.port and the camera names/indices are specific to your machine, and the camera names must match the observation keys this policy was trained on.

When --strategy.type=base is used the script doesn't record the episodes. Skipping duration will make the policy run indefinitely. For more information look at rollout documentation.

Train your own policy

lerobot-train \
  --dataset.repo_id=${HF_USER}/<dataset> \
  --policy.type=molmoact2 \
  --output_dir=outputs/train/<policy_repo_id> \
  --job_name=lerobot_training \
  --policy.device=cuda \
  --policy.repo_id=${HF_USER}/<policy_repo_id> \
  --wandb.enable=true

Writes checkpoints to outputs/train/<policy_repo_id>/checkpoints/.

Evaluation

No evaluation results have been provided for this policy yet.

Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}

Downloads last month: -

Safetensors

Model size

5B params

Tensor type

F32

BF16

Video Preview

Robotics

justintiensmith
/

molmoact2_VLA_Reasoning_Training_Dataset_1200