Phongwit/act_crane_k5

ACT policy for CRANE+ V2 pick-and-place (Gazebo)

An ACT (Action Chunking with Transformers) policy trained with LeRobot on <your-hf-name>/crane_pick_place. The task is to pick up a box from the table and place it on a toy car, both at random positions, in a Gazebo simulation of a CRANE+ V2 arm.

Code: github.com/bamboo51/Arm-Robot-Picking

Status: research checkpoint, not a working policy. Offline, it reproduces the expert's motion on training scenes. In Gazebo, it does not yet turn the base toward the box reliably. See Results.

Inputs and outputs

Input	3 RGB cameras (top, left, right; 240×320) and 5 joint angles (4 arm joints and 1 gripper joint, rad)
Output	target joint angles 0.5 s ahead (5 values)
Control rate	10 Hz

Training

Setting	Value
Policy	act (LeRobot defaults otherwise)
chunk_size	50
n_action_steps	1 (the policy re-plans every step)
Batch size	16
Steps	20,000 (about 20 epochs)
Final loss	0.055 (L1 0.050, KLD 0.001)
Hardware	1× RTX 4080, about 1 h 12 min

Results

Offline: predicted vs expert actions

These numbers come from one training episode and one held-out test episode. The hold baseline is the error of predicting "stay at the current joint angles". A policy that has learned the motion should do better than this baseline.

	Mean joint error	Hold baseline	Gripper error
Training episode	0.037 rad	0.061 rad	0.012 rad
Test episode	0.067 rad	0.066 rad	0.020 rad

• Training episode: the policy beats the baseline from about step 8,000 onward.
• Test episode: it only reaches the baseline at step 20,000, and the test error was still falling at that point.
• Gripper timing: it generalizes well on both episodes.

Gazebo (closed loop)

• What works: the arm leaves the start pose, and joints 2–4 move into the reach-down shape.
• What fails: the base rotation (joint1) only creeps, so the arm does not aim at the box.
  • The likely cause is too few examples per box angle (51 episodes over a 260° range). The policy predicts roughly the average base angle instead of reading the box's direction from the images.

Usage

The checkpoint is served over ZMQ to a ROS 2 node that reads the cameras and publishes joint trajectories:

# policy server (Python with LeRobot, GPU)
python learning/policy_server.py <path or repo id of this model>
# ROS 2 node (with the Gazebo simulation running)
python3 scripts/infer_node.py

To load the policy directly in Python:

from lerobot.policies.act.modeling_act import ACTPolicy
policy = ACTPolicy.from_pretrained("Phongwit/act_crane_k5")

Next steps

• Narrow the box angle range, or collect more episodes per angle.
• Train for longer, since the test error was still decreasing at 20,000 steps.
• Try the top camera only.