lichtphyz/act_v3_aug_desktop

ACT: SO-101 block to bin

An Action Chunking with Transformers (ACT) policy for a single SO-101 arm that picks up a wooden block and drops it into a bin, using a front (workspace) camera and a wrist camera. Trained with LeRobot on 125 teleoperated episodes, with image augmentation.

Part of the SO-101 block-to-bin collection (models, training data and real-robot evaluations).

Real-robot result: 9/10 successes with temporal ensembling (see Evaluation). All 10 evaluation episodes, including camera video, are public: lichtphyz/rollout_act_v3_aug_desktop.

Run it with temporal ensembling (--policy.temporal_ensemble_coeff=0.01 --policy.n_action_steps=1). The trained default executes each 100-step chunk (3.3 s) open-loop, which missed grasps often in earlier versions of this model. Short chunks without ensembling (e.g. n_action_steps=10) make the arm sit still at its rest pose, because the demonstrations start with about 2 s of stillness.

Model Details

• License: apache-2.0
• Robot: SO-101 follower (so_follower), 6 joints (5 + gripper), STS3215 12V servos
• Cameras: front (fixed workspace view) and wrist, both 640×480 @ 30 fps
• Task string: "Put the block in the bin."
• Sibling model: lichtphyz/diffusion_v3_desktop, a Diffusion Policy on the same data (8/10).

Inputs & Outputs

The policy consumes these observation features and produces these action features.

Inputs

Feature	Type	Shape
observation.state	STATE	(6,)
observation.images.front	VISUAL	(3, 480, 640)
observation.images.wrist	VISUAL	(3, 480, 640)

Outputs

Feature	Type	Shape
action	ACTION	(6,)

Training Dataset

• Repository: lichtphyz/so101_block_to_bin_v3
• Episodes: 125
• Frames: 55677
• Frame rate: 30 FPS
• Task(s): "Put the block in the bin."

The 125 training episodes are 100 block-to-bin demonstrations (the block grid-aligned, two block lengths, hover-then-descend approach), 15 "idle" episodes (empty table or block already in the bin: the arm stays at rest), and 10 recovery episodes (missed grasp → regrasp → bin).

Training Configuration

Setting	Value
Training steps	80,000 (on one RTX 4080)
Batch size	8
Optimizer / LR	AdamW, 1e-5 (backbone 1e-5), constant
Vision backbone	ResNet-18 (ImageNet weights)
Transformer	512-dim, 4 encoder layers / 1 decoder layer, VAE (latent 32, KL weight 10)
Chunk size	100 actions (3.3 s at 30 fps)
Image augmentation	brightness, contrast, saturation, hue, sharpness, small affine
Seed	1000
LeRobot version	0.6.2

Apart from image augmentation, the settings match an earlier ACT run trained on the same data on a laptop.

---

How to Get Started with the Model

Run it on an SO-101 with the settings used in the evaluation:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=lichtphyz/act_v3_aug_desktop \
  --policy.temporal_ensemble_coeff=0.01 \
  --policy.n_action_steps=1 \
  --robot.type=so101_follower \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ front: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}, wrist: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}}" \
  --task="Put the block in the bin." \
  --duration=60

The camera names must be front and wrist. The policy will only work with a camera placement and workspace close to the training data; see the dataset visualizer links above.

---

Evaluation

Real-robot evaluation with lerobot-rollout --strategy.type=episodic: 10 episodes of up to 30 s each, the block reset by hand between episodes, inference with temporal ensembling (temporal_ensemble_coeff=0.01, n_action_steps=1).

Task	Trials	Successes	Success rate
Put the block in the bin	10	9	90%

• The one failure was early in the session (episode 1, 2 or 3).
• Two successes bumped the bin edge with the block, because the arm didn't lift high enough and/or the grip was off-center (but firm); the block still went in. These were the same two placements on which the Diffusion Policy failed (its evaluation: 8/10).
• Grasps were usually more off-center than the Diffusion Policy's, but the policy visibly adjusted toward the block's center, and the grip was good enough in all but one case.
• Both block sizes. The training data has two block lengths. In an extra, unrecorded test with the shorter block, this policy succeeded 4/4; the Diffusion Policy had some trouble with short blocks.
• With 10 trials each, 9/10 vs 8/10 is not a meaningful difference between the two models.
• All 10 episodes (both camera streams, joint states and actions) are in lichtphyz/rollout_act_v3_aug_desktop.

---

Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}