Frozen-Backbone Residual RL for MolmoAct2 — trained residual checkpoints
Residual-policy checkpoints for the RL4VLA @ RSS 2026 workshop paper "Hill-Climbing a Real-World VLA in Simulation: Frozen-Backbone Residual Reinforcement Learning for MolmoAct2" (Atharva Kshirsagar, Tirth Gada, Vrushtee Gaikwad).
The backbone (allenai/MolmoAct2-BimanualYAM) stays frozen; each checkpoint
here is a small residual actor + twin critic (2×256 MLP, ~1 MB) that adds a
bounded correction (α=0.05) to the backbone's decoded 14-D action chunk on
BimanualYAMPutEverythingInBoxMiscal2RL-v1 (ManiSkill, 0.02 rad actuator
miscalibration, 1500-step horizon).
Checkpoints
| Path | Learner | Held-out success | n |
|---|---|---|---|
td3bc_seed1/actor_final.pt |
TD3+BC | 20% | 30 |
td3bc_seed2/actor_final.pt |
TD3+BC | 13% (= base) | 30 |
sacbc_seed0/actor_final.pt |
SAC+BC | 33% | 45 |
Frozen zero-shot base: 13%. Naive (un-anchored) TD3 collapses to 0%.
The paper's seed-0 TD3+BC checkpoint (33%, n=15) predates this repo and was
not preserved. Note: single 15-episode evals carry ~±12pp noise (the flow
decoder samples at inference) — success rates above pool 2–3 eval passes.
Each directory includes the training config.json and per-episode
metrics.jsonl; evals/ holds the raw held-out eval JSONs.
Usage
Code (a drop-in extension of the public MolmoAct2 sim_eval stack):
https://github.com/vovw/rl4vla-residual-rl
uv run python -m sim_eval.rl.eval_residual_yam \
--env-id BimanualYAMPutEverythingInBoxMiscal2RL-v1 \
--checkpoint sacbc_seed0/actor_final.pt --algo sac \
--n-episodes 15 --max-episode-steps 1500 --residual-scale-override 0.05