MolmoAct2-LIT and MolmoAct2-baseline (Bimanual YAM)

Two MolmoAct2 vision-language-action policies for a bimanual YAM robot. Both are trained on identical data with an identical optimization budget and differ in exactly one respect: the presence of a goal-pose prior. They are published together because the pair constitutes a controlled comparison.

	Molmoact2-LIT/	Molmoact2-baseline/
Goal-pose prior	yes	no
Stage-1 pose pretraining	yes	no
Goal conditioning	semantic_visual_recurrent	none
Total parameters	5,615,802,112	5,442,196,208
Trainable parameters	5,219,636,992	5,046,031,088

The two models differ by 173,605,904 parameters (3.2%), which is the semantic-visual module and nothing else.

Architectures

Molmoact2-baseline/

Standard MolmoAct2 continuous-control fine-tuning. enable_goal_pose=false, so the model contains no SE(3) goal encoder, no learnable goal queries, no semantic-visual module and no pose-reconstruction objective; the training loss is flow matching only. The action expert attends to image tokens directly (mask_image_from_action_expert=false).

Molmoact2-LIT/

Trained in two stages.

Stage 1 — pose prior without vision. Only the action expert and a 5.5M-parameter SE(3) goal encoder are trainable (583,088,928 parameters, 10.7% of the model). The VLM, ViT and connector are frozen, and disable_visual_input=true means the vision tower does not run. The model learns to produce a trajectory between the current pose and a target pose 30 frames ahead.

Stage 2 — visual steering. All parameters unfreeze. The SE(3) encoder is replaced by goal_token_source=learnable_queries feeding a semantic_visual_recurrent module, which infers the goal pose from images and language rather than receiving it directly. Raw image key-values reach the action expert only through this latent aggregator (mask_image_from_action_expert=true). The pose-reconstruction term is a training signal only; at inference no goal pose is supplied.

Training data

A bimanual YAM dataset covering three tasks.

Episodes	292
Frames	283,886
Training anchors	275,067
Frame rate	30 fps
Cameras	top (360×640), left (480×640), right (480×640)
Instructions	Put all blocks into the box. · Clean the table using the dust pan. · Transfer the egg from the pan into the bowl.

Both models use the same dataset and the same anchor manifest, so at a fixed seed they observe identical samples in identical order.

Hyperparameters

Shared — both models

Parameter	Value
Training steps	6,400
Global batch size	384 (96 per GPU × 4 GPUs × 1 accumulation step)
Samples seen	2,457,600 (8.9 epochs)
Optimizer	AdamW
Learning rate — VLM	1e-4
Learning rate — ViT	1e-4
Learning rate — connector	1e-4
Learning rate — action expert	3e-4
Betas	[0.9, 0.95]
Epsilon	1e-6
Weight decay	0.0
Gradient clip norm	1.0
LR schedule	cosine, floor 1e-5, horizon 17,600 steps
Warmup — VLM / connector / action expert	1,000 steps
Warmup — ViT	2,000 steps
Precision	bfloat16
Gradient checkpointing	enabled
Embeddings	frozen
Flow-matching timesteps (training)	8
Denoising steps (inference)	10
Max sequence length	896
Action chunk / horizon	30 (1.0 s at 30 fps)
Seed	1000
Hardware	4 × NVIDIA H200

Molmoact2-LIT/ — additional

Parameter	Value
Stage-1 steps	10,000
Stage-1 global batch	1,024 (256 × 4)
Stage-1 learning rate (action expert, goal encoder)	5e-5, cosine to 1e-6
Stage-1 trainable parameters	583,088,928 (10.7%)
goal_token_source	learnable_queries
goal_conditioning_mode	semantic_visual_recurrent
Latent tokens	100 (8 pose + 92 context)
Layer groups	6
Hidden dim / heads / FFN ratio	768 / 8 / 4.0
Learning rate — semantic-visual module	3e-4
Warmup — semantic-visual module	1,000 steps
Goal horizon (target_pose_delta_index)	30 frames
Pose-reconstruction loss weight	0.3
mask_image_from_action_expert	true

Initialization

Identical for both models: the released allenai/MolmoAct2 checkpoint (revision e432d85f6e039edca44afb93c262f3084ab72a9c) as a template, its VLM tensors replaced by allenai/Molmo2-ER (revision dab22564403d2607855bb1fffb0721285b445081; 704 of 706 tensors loaded exactly), with the action expert randomly re-initialized. Neither model inherits released MolmoAct2 action-expert weights.

Input / output contract

Both models share one schema.

Cameras	3 RGB streams in order top, left, right
State / action dimension	16
Action space	absolute end-effector pose (not deltas)
Layout	[x, y, z, qw, qx, qy, qz, gripper] × 2 arms; left 0:8, right 8:16
Quaternion	w-first, at indices 3, 7 (left) and 11, 15 (right)
Gripper	dimensions 7 and 15, unnormalized (normalize_gripper=false)
Chunk	30 actions = 1.0 s at 30 fps
Normalization	quantile-based, q01/q99 → [-1, 1]

Normalization statistics ship with each model in the preprocessor and postprocessor safetensors files.

Usage

from lerobot.policies.molmoact2.modeling_molmoact2 import MolmoAct2Policy

policy = MolmoAct2Policy.from_pretrained(
    "shailes-h/Molmoact2-LIT", subfolder="Molmoact2-LIT"
)
# or  subfolder="Molmoact2-baseline"

To download a single model:

hf download shailes-h/Molmoact2-LIT --include "Molmoact2-LIT/*" --local-dir ./lit

Each folder contains config.json, model.safetensors, the preprocessor and postprocessor configurations with their statistics, and train_config.json recording the full run configuration. Optimizer state is not included; these are inference artifacts.