LamportRobotics/imago-pack-can
Two small networks that pack a can into a tote with Imago's right arm, in simulation, from the robot's own camera. Trained by behaviour cloning on scripted demonstrations.
The hand (pack_can.onnx): joint positions and the can's position in, the next 25 joint targets
out. An MLP, 3 x 512. At run time the first 10 targets of each chunk are played at 50 Hz, then it
is asked again.
- input
obs[B, 17]: 14 joint positions (rad) then the can's x, y, z (m, world frame) - output
targets[B, 25, 14]: joint position targets (rad), 20 ms apart
The eye (eye.onnx): the visor camera picture and the joints in, where the can is out. A
4-layer conv with spatial soft-argmax keypoints, then an MLP. It needs the joints because the
camera turns with the waist.
- input
image[B, 128, 128, 3]: RGB, 0..255 - input
state[B, 14]: joint positions (rad) - output
can_xyz[B, 3]: the can's position (m, world frame)
Together: the eye looks once at the start, the hand runs on its answer. Learning targets straight from pixels was tried first on the same data and packed 36 of 100; the split is what is here.
Normalisation is inside both graphs. Nothing else is needed to run them in onnxruntime.
Results, on cans the policies had not seen
| trials | expert | policy | |
|---|---|---|---|
| hand, told where the can is | 100 | 99 | 100 |
| eye + hand, from the camera | 100 | 99 | 99 |
Eye error on held-out pictures: 0.8 / 1.5 / 0.6 (x / y / z) mm. The hand was trained on 980 demonstrations, the eye on 980.
Joints, in order: waist_yaw, right_shoulder_pitch, right_shoulder_roll, right_shoulder_yaw, right_elbow, right_wrist_roll, right_wrist_flex, right_wrist_dev, right_index, right_middle, right_ring, right_little, right_thumb_swing, right_thumb.
The robot
Imago is Lamport's humanoid, a 1.50 m, 48 kg simulation model with five-finger hands, built in MuJoCo. It exists only in simulation. The task uses 14 of its 42 driven joints: waist yaw, the right arm (7) and the right hand (6). Joint zero is the standing pose.
Files
pack_can.onnx,eye.onnx: the two networks, self-containedpack_can.pt,eye.pt: PyTorch checkpoints with the normalisation statisticsbc_result.json,bc_cam_result.json: the evaluation runs abovepolicy.gif,policy_cam.gif: the policy at work
Data: the datasets listed above. Simulation only; nothing here has run on hardware.