| --- |
| datasets: MadmanMischief/so101_candy_sort_sim_half |
| library_name: lerobot |
| model_name: act |
| pipeline_tag: robotics |
| tags: |
| - robotics |
| - act |
| - lerobot |
| --- |
| |
| # ACT, 55 sim demos only — SO101 candy sort |
|
|
| **Task:** teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: `"pick up the blue candy"`. Built during the [B1S Hacktober Physical AI 2026](https://www.b1s.ai/hackathons/b1s-hacktober-physical-ai-2026) hackathon with LeRobot 0.6.2. |
|
|
| **Robot:** SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: `wrist` (USB webcam on the gripper) and `phone` (Android phone beside the table). |
|
|
| ## Training |
|
|
| - **Dataset:** [MadmanMischief/so101_candy_sort_sim_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_sim_half) |
| - ACT on the L40S, batch 16, 50k steps, on 55 scripted pick-and-place demos recorded in the Isaac Sim twin. Repo root = 50k. |
|
|
| ## Results |
|
|
| **Real arm** |
|
|
| - Not run on the arm. |
|
|
| **Sim (digital twin)** |
|
|
| - 50k: 0 / 5 (0 / 10), never lifted a blue candy. |
|
|
| Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp. |
|
|
| ## Run it with LeRobot |
|
|
| ```bash |
| lerobot-rollout \ |
| --strategy.type=base \ |
| --policy.path=MadmanMischief/act_candy_sort_sim \ |
| --robot.type=so101_follower \ |
| --robot.port=/dev/ttyACM0 \ |
| --robot.id=my_follower_arm \ |
| --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \ |
| --task="pick up the blue candy" \ |
| --duration=40 |
| ``` |
| The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a `downscale: 2` option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under `checkpoints/<step>/pretrained_model` (download with `hf download MadmanMischief/act_candy_sort_sim --include 'checkpoints/<step>/*'`). |
|
|
| ## Inputs & Outputs |
|
|
| The policy consumes these observation features and produces these action features. |
|
|
| **Inputs** |
|
|
| | Feature | Type | Shape | |
| | --- | --- | --- | |
| | `observation.state` | STATE | `(6,)` | |
| | `observation.images.wrist` | VISUAL | `(3, 240, 320)` | |
| | `observation.images.phone` | VISUAL | `(3, 360, 640)` | |
|
|
| **Outputs** |
|
|
| | Feature | Type | Shape | |
| | --- | --- | --- | |
| | `action` | ACTION | `(6,)` | |
|
|
| ## Related |
|
|
| Datasets: [so101_candy_sort](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort) · [so101_candy_sort_half_127](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_half_127) · [so101_candy_sort_sim_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_sim_half) · [so101_candy_sort_mix_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_mix_half) · [so101_candy_grid](https://huggingface.co/datasets/MadmanMischief/so101_candy_grid). RL policy and digital twin: [so101_candy_rl](https://huggingface.co/MadmanMischief/so101_candy_rl). |
|
|
