RoboColosseum/yam-dustpan-molmoact2

MolmoAct2 — BimanualYAM Dustpan

• Base: allenai/MolmoAct2, molmoact2 repo, train_lerobot.py launch script
• Data: BimanualYAM Dustpan lerobot dataset, 14-D absolute joint state/action, cameras top/left/right
• Trained: full VLM backbone + action expert (ft_vlm=true → ViT, connector, and LLM all trainable; ft_action_expert=true; ft_embedding=lm_head), AdamW, no LoRA
• Optim: LLM LR 2e-5, ViT/connector LR 1e-5, action expert LR 1e-4, warmup 100-200 steps (per-component), device batch 32 → GBS 128, 5 epochs, ~4h37m
• Loss: action_flow_loss 0.049 → 0.007 (flow-matching loss on the action expert; no language-modeling loss term is logged for this run since the lerobot dataset carries no text-labeled examples — see note below)
• Files: HF-format export (model-*.safetensors, config.json, processor_config.json, tokenizer.json, remote-code files, norm_stats.json), converted from the native sharded training checkpoint via experiments/olmo/hf_model/convert_molmoact2_to_hf.py
• wandb: https://wandb.ai/shaileshxml-nus/RoboColosseum/runs/qecaa9le

Note on loss curve: ft_vlm=true does make the ViT/connector/LLM trainable, but since train_lerobot.py only supplies image+instruction+action batches (no text/VQA labels), the language-modeling loss branch never fires — gradients reach the VLM backbone only through the action flow-matching loss. This is expected behavior for this training pipeline, not a bug.