---
datasets: MadmanMischief/so101_candy_grid
library_name: lerobot
model_name: act
pipeline_tag: robotics
tags:
- act
- robotics
- lerobot
---

# ACT, 71 grid demos (new camera view) — SO101 candy sort

**Task:** teach an SO101 arm (leader/follower teleoperation) to pick the blue candies out of a mixed blue and green pile and drop them on a towel. Language prompt: `"pick up the blue candy"`. Built during the [B1S Hacktober Physical AI 2026](https://www.b1s.ai/hackathons/b1s-hacktober-physical-ai-2026) hackathon with LeRobot 0.6.2.

**Robot:** SO101 follower, 6 joints (pan, lift, elbow, wrist flex, wrist roll, gripper) at 30 Hz. Cameras: `wrist` (USB webcam on the gripper) and `phone` (Android phone beside the table).

## Training

- **Dataset:** [MadmanMischief/so101_candy_grid](https://huggingface.co/datasets/MadmanMischief/so101_candy_grid)
- ACT on the L40S, batch 16, 50k steps, on 71 demos with candies spread on a taped 3x4 grid and the phone in a new position. Checkpoints every 7.5k; repo root = 50k.

## Results

**Real arm**

- 7.5k, 15k, 30k and 37.5k were run on the arm (9 recordings), outcomes not scored.

**Sim (digital twin)**

- Not run.

Sim scores are out of 5 rollouts (candy layouts seeds 500 to 504) in an Isaac Sim 6.1 / Isaac Lab digital twin; a rollout succeeds when a blue candy ends on the towel within 20 s. The twin uses assisted grasp.

## Run it with LeRobot

```bash
lerobot-rollout \
  --strategy.type=base \
  --policy.path=MadmanMischief/act_candy_sort_grid \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=my_follower_arm \
  --robot.cameras="{ wrist: {type: opencv, index_or_path: /dev/video4, width: 640, height: 480, fps: 30}, phone: {type: opencv, index_or_path: <phone camera>, width: 640, height: 360, fps: 30} }" \
  --task="pick up the blue candy" \
  --duration=40
```
The policy expects the image sizes listed under Inputs below (half resolution: wrist 320x240, phone 640x360; we added a `downscale: 2` option to LeRobot's OpenCV camera for the wrist cam). Other checkpoints are under `checkpoints/<step>/pretrained_model` (download with `hf download MadmanMischief/act_candy_sort_grid --include 'checkpoints/<step>/*'`).

## Inputs & Outputs

The policy consumes these observation features and produces these action features.

**Inputs**

| Feature | Type | Shape |
| --- | --- | --- |
| `observation.state` | STATE | `(6,)` |
| `observation.images.wrist` | VISUAL | `(3, 240, 320)` |
| `observation.images.phone` | VISUAL | `(3, 360, 640)` |

**Outputs**

| Feature | Type | Shape |
| --- | --- | --- |
| `action` | ACTION | `(6,)` |

## Related

Datasets: [so101_candy_sort](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort) · [so101_candy_sort_half_127](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_half_127) · [so101_candy_sort_sim_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_sim_half) · [so101_candy_sort_mix_half](https://huggingface.co/datasets/MadmanMischief/so101_candy_sort_mix_half) · [so101_candy_grid](https://huggingface.co/datasets/MadmanMischief/so101_candy_grid). RL policy and digital twin: [so101_candy_rl](https://huggingface.co/MadmanMischief/so101_candy_rl).