Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware
Paper • 2304.13705 • Published • 7
ACT — SO101 pick-and-place, sim-pretrained + real co-trained
An ACT policy for the low-cost SO-101 arm that picks up a 4 cm cube and places it on a white sheet. It is pretrained entirely in simulation (Genesis) with domain randomization, then fine-tuned on a mixture that is 33% real teleoperation data.
Trained on an AMD Radeon GPU (ROCm) for the AMD AI DevMaster Hackathon, Track 3 (Physical AI).
Training recipe
| Stage | Data | Steps | Result |
|---|---|---|---|
| Sim pretrain | so101_cube_dr — 50 scripted episodes, full DR (cube colour, table, lighting, friction, mass) |
60k | base checkpoint |
| Co-train fine-tune | so101_mixed — 10 real teleop episodes repeated ×3 (30) + 50 sim (80 total, 66,539 frames, 33.1% real) |
20k | this checkpoint |
- ACT: chunk size 100, hidden dim 512, ResNet18 ×2 cameras, VAE
- batch 8, lr 1e-5, image transforms enabled
- fine-tune ran 2 h 22 m at 2.54 step/s, 3.9 GB VRAM, final loss 0.065 over 2.40 epochs
torch 2.9.1+rocm7.2.1
I/O
observation.state— 6 joint positions, radians (5 arm joints + gripper jaw angle)observation.images.world,observation.images.wrist— 640×480 RGB, 30 fpsaction— 6 commanded joint positions, radians
The real SO-101 driver in LeRobot reports arm joints in degrees and the gripper in
RANGE_0_100, so a conversion layer is required on hardware. Gripper convention here:
jaw open ≈ 1.3 rad, closed ≈ −0.05 rad.
Sibling checkpoints
- 10k-step fine-tune of the same run:
checkpoint-10000/in this repo - The lineage's sim-only ancestors and an ablation trained on the 10 real episodes alone are described in the project report.
Status and limitations
- Simulation: the red-only sibling of this lineage peaks at 60% success at 60k steps on a fixed-seed 20-episode evaluation. Success saturates well before the loss does — a data ceiling of the 50-episode sim set, not an optimisation failure.
- Real robot: 85% success (17/20), placing the cube within 1-2 cm of the target centre, evaluated over 20 fixed placements. The same 10 real episodes trained without the simulated pretraining reach 25% (5/20) on the identical placements — Fisher exact p = 0.00033. The simulated pretraining is what makes 10 real demonstrations enough.
- Sim-only checkpoints transfer visually (the arm reaches the real cube from real camera images) but do not complete the grasp on their own. Independently measured contributors to that gap: simulated PD gain KP=120 versus the servo's P=16, the wrist camera's unmodelled mass, and a 311 ms chunk-replan latency that simulation hides because it waits for computation.
- Trained for one object (a 4 cm cube) on one table, from a single camera rig. Colour generalisation follows object–background RGB contrast rather than hue: success correlates 0.956 with ‖Δc‖ and collapses when the cube matches the table colour.
Related
- Downloads last month
- 18