Robotics
ONNX
imago
humanoid
simulation
mujoco
behavior-cloning
lamport-lab

LamportRobotics/imago-pack-can

Two small networks that pack a can into a tote with Imago's right arm, in simulation, from the robot's own camera. Trained by behaviour cloning on scripted demonstrations.

The hand (pack_can.onnx): joint positions and the can's position in, the next 25 joint targets out. An MLP, 3 x 512. At run time the first 10 targets of each chunk are played at 50 Hz, then it is asked again.

  • input obs [B, 17]: 14 joint positions (rad) then the can's x, y, z (m, world frame)
  • output targets [B, 25, 14]: joint position targets (rad), 20 ms apart

The eye (eye.onnx): the visor camera picture and the joints in, where the can is out. A 4-layer conv with spatial soft-argmax keypoints, then an MLP. It needs the joints because the camera turns with the waist.

  • input image [B, 128, 128, 3]: RGB, 0..255
  • input state [B, 14]: joint positions (rad)
  • output can_xyz [B, 3]: the can's position (m, world frame)

Together: the eye looks once at the start, the hand runs on its answer. Learning targets straight from pixels was tried first on the same data and packed 36 of 100; the split is what is here.

Normalisation is inside both graphs. Nothing else is needed to run them in onnxruntime.

Results, on cans the policies had not seen

trials expert policy
hand, told where the can is 100 99 100
eye + hand, from the camera 100 99 99

Eye error on held-out pictures: 0.8 / 1.5 / 0.6 (x / y / z) mm. The hand was trained on 980 demonstrations, the eye on 980.

Joints, in order: waist_yaw, right_shoulder_pitch, right_shoulder_roll, right_shoulder_yaw, right_elbow, right_wrist_roll, right_wrist_flex, right_wrist_dev, right_index, right_middle, right_ring, right_little, right_thumb_swing, right_thumb.

The robot

Imago is Lamport's humanoid, a 1.50 m, 48 kg simulation model with five-finger hands, built in MuJoCo. It exists only in simulation. The task uses 14 of its 42 driven joints: waist yaw, the right arm (7) and the right hand (6). Joint zero is the standing pose.

Files

  • pack_can.onnx, eye.onnx: the two networks, self-contained
  • pack_can.pt, eye.pt: PyTorch checkpoints with the normalisation statistics
  • bc_result.json, bc_cam_result.json: the evaluation runs above
  • policy.gif, policy_cam.gif: the policy at work

Data: the datasets listed above. Simulation only; nothing here has run on hardware.

Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Datasets used to train LamportRobotics/imago-pack-can