File size: 3,183 Bytes
38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc 38454bf 41a6cbc | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 | ---
datasets: MadmanMischief/so101_candy_sort_sim_half
library_name: lerobot
model_name: act
pipeline_tag: robotics
tags:
- robotics
- act
- lerobot
---
# ACT, 55 sim demos only — SO101 candy sort
**Task:** teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: `"pick up the blue candy"`. Built during the [B1S Hacktober Physical AI 2026](https://www.b1s.ai/hackathons/b1s-hacktober-physical-ai-2026) hackathon with LeRobot 0.6.2.
**Robot:** SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: `wrist` (USB webcam on the gripper) and `phone` (Android phone beside the table).
## Training
- **Dataset:** [MadmanMischief/so101_candy_sort_sim_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_sim_half)
- ACT on the L40S, batch 16, 50k steps, on 55 scripted pick-and-place demos recorded in the Isaac Sim twin. Repo root = 50k.
## Results
**Real arm**
- Not run on the arm.
**Sim (digital twin)**
- 50k: 0 / 5 (0 / 10), never lifted a blue candy.
Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.
## Run it with LeRobot
```bash
lerobot-rollout \
--strategy.type=base \
--policy.path=MadmanMischief/act_candy_sort_sim \
--robot.type=so101_follower \
--robot.port=/dev/ttyACM0 \
--robot.id=my_follower_arm \
--robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
--task="pick up the blue candy" \
--duration=40
```
The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a `downscale: 2` option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under `checkpoints/<step>/pretrained_model` (download with `hf download MadmanMischief/act_candy_sort_sim --include 'checkpoints/<step>/*'`).
## Inputs & Outputs
The policy consumes these observation features and produces these action features.
**Inputs**
| Feature | Type | Shape |
| --- | --- | --- |
| `observation.state` | STATE | `(6,)` |
| `observation.images.wrist` | VISUAL | `(3, 240, 320)` |
| `observation.images.phone` | VISUAL | `(3, 360, 640)` |
**Outputs**
| Feature | Type | Shape |
| --- | --- | --- |
| `action` | ACTION | `(6,)` |
## Related
Datasets: [so101_candy_sort](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort) · [so101_candy_sort_half_127](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_half_127) · [so101_candy_sort_sim_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_sim_half) · [so101_candy_sort_mix_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_mix_half) · [so101_candy_grid](https://huggingface.co/datasets/MadmanMischief/so101_candy_grid). RL policy and digital twin: [so101_candy_rl](https://huggingface.co/MadmanMischief/so101_candy_rl).
|