datasets: MadmanMischief/so101_candy_sort_sim_half
library_name: lerobot
model_name: act
pipeline_tag: robotics
tags:
- robotics
- act
- lerobot
ACT, 55 sim demos only — SO101 candy sort
Task: teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: "pick up the blue candy". Built during the B1S Hacktober Physical AI 2026 hackathon with LeRobot 0.6.2.
Robot: SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: wrist (USB webcam on the gripper) and phone (Android phone beside the table).
Training
- Dataset: MadmanMischief/so101_candy_sort_sim_half
- ACT on the L40S, batch 16, 50k steps, on 55 scripted pick-and-place demos recorded in the Isaac Sim twin. Repo root = 50k.
Results
Real arm
- Not run on the arm.
Sim (digital twin)
- 50k: 0 / 5 (0 / 10), never lifted a blue candy.
Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.
Run it with LeRobot
lerobot-rollout \
--strategy.type=base \
--policy.path=MadmanMischief/act_candy_sort_sim \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_follower_arm \
--robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
--task="pick up the blue candy" \
--duration=40
The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a downscale: 2 option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under checkpoints/<step>/pretrained_model (download with hf download MadmanMischief/act_candy_sort_sim --include 'checkpoints/<step>/*').
Inputs & Outputs
The policy consumes these observation features and produces these action features.
Inputs
| Feature | Type | Shape |
|---|---|---|
observation.state |
STATE | (6,) |
observation.images.wrist |
VISUAL | (3, 240, 320) |
observation.images.phone |
VISUAL | (3, 360, 640) |
Outputs
| Feature | Type | Shape |
|---|---|---|
action |
ACTION | (6,) |
Related
Datasets: so101_candy_sort · so101_candy_sort_half_127 · so101_candy_sort_sim_half · so101_candy_sort_mix_half · so101_candy_grid. RL policy and digital twin: so101_candy_rl.