File size: 6,536 Bytes

---
datasets:
- lichtphyz/so101_block_to_bin_v3
- lichtphyz/rollout_diffusion_v3_desktop
library_name: lerobot
license: apache-2.0
model_name: diffusion
pipeline_tag: robotics
tags:
- robotics
- lerobot
- diffusion
- so101
---

# Diffusion Policy: SO-101 block to bin

A [Diffusion Policy](https://huggingface.co/papers/2303.04137) for a single **SO-101** arm that picks up a wooden block
and drops it into a bin, using a front (workspace) camera and a wrist camera. Trained with
[LeRobot](https://github.com/huggingface/lerobot) on 125 teleoperated episodes.

Part of the [SO-101 block-to-bin collection](https://huggingface.co/collections/lichtphyz/so-101-block-to-bin-act-vs-diffusion-policy-6ad6ca5c5e111b426bb1f8a6) (models, training data and real-robot evaluations).

**Real-robot result: 8/10 successes** (DDIM, 5 denoising steps); see [Evaluation](#evaluation).
All 10 evaluation episodes, including camera video, are public:
[lichtphyz/rollout_diffusion_v3_desktop](https://huggingface.co/spaces/lerobot/visualize_dataset?path=lichtphyz/rollout_diffusion_v3_desktop).

## Model Details

- **License:** apache-2.0
- **Robot:** SO-101 follower (`so_follower`), 6 joints (5 + gripper), STS3215 12V servos
- **Cameras:** `front` (fixed workspace view) and `wrist`, both 640×480 @ 30 fps
- **Task string:** "Put the block in the bin."
- **Sibling model:** [lichtphyz/act_v3_aug_desktop](https://huggingface.co/lichtphyz/act_v3_aug_desktop), an ACT
  policy on the same data (9/10).

## Inputs & Outputs

The policy consumes these observation features and produces these action features.

**Inputs**

| Feature | Type | Shape |
| --- | --- | --- |
| `observation.state` | STATE | `(6,)` |
| `observation.images.front` | VISUAL | `(3, 480, 640)` |
| `observation.images.wrist` | VISUAL | `(3, 480, 640)` |

**Outputs**

| Feature | Type | Shape |
| --- | --- | --- |
| `action` | ACTION | `(6,)` |


## Training Dataset

- **Repository:** [lichtphyz/so101_block_to_bin_v3](https://huggingface.co/datasets/lichtphyz/so101_block_to_bin_v3)
- **Episodes:** 125
- **Frames:** 55677
- **Frame rate:** 30 FPS
- **Task(s):** "Put the block in the bin."

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=lichtphyz/so101_block_to_bin_v3">
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
</a>


The 125 training episodes are 100 block-to-bin demonstrations (the block grid-aligned, two block lengths,
hover-then-descend approach), 15 "idle" episodes (empty table or block already in the bin: the arm stays at rest), and
10 recovery episodes (missed grasp → regrasp → bin).

## Training Configuration

| Setting | Value |
| --- | --- |
| Training steps | 60,000 (≈11 h 20 min on one RTX 4080) |
| Batch size | 32 |
| Optimizer / LR | Adam, 1e-4, cosine schedule |
| Vision backbone | ResNet-18 (ImageNet weights, BatchNorm) |
| Image size | resized to 320×240, random crop 304×228 (crop ratio 0.95) |
| Observation steps / horizon / action steps | 2 / 64 / 32 |
| Training noise schedule | DDPM, 100 timesteps |
| Image augmentation | brightness, contrast, saturation, hue, sharpness, small affine |
| Seed | 1000 |
| LeRobot version | 0.6.2 |

---

## How to Get Started with the Model

Run it on an SO-101 with the settings used in the evaluation (DDIM with 5 denoising steps makes action planning fast
enough to avoid visible pauses):

```bash
lerobot-rollout \
  --strategy.type=base \
  --policy.path=lichtphyz/diffusion_v3_desktop \
  --policy.noise_scheduler_type=DDIM \
  --policy.num_inference_steps=5 \
  --robot.type=so101_follower \
  --robot.port=<your_robot_port> \
  --robot.cameras="{ front: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}, wrist: {type: opencv, index_or_path: <path>, width: 640, height: 480, fps: 30, fourcc: MJPG}}" \
  --task="Put the block in the bin." \
  --duration=60
```

The camera names must be `front` and `wrist`. The policy will only work with a camera placement and workspace close to
the training data; see the dataset visualizer links above.

---

## Evaluation

Real-robot evaluation with `lerobot-rollout --strategy.type=episodic`: 10 episodes of up to 30 s each, the block reset
by hand between episodes, inference with DDIM and 5 denoising steps.

| Task | Trials | Successes | Success rate |
| ---- | ------ | --------- | ------------ |
| Put the block in the bin | 10 | 8 | 80% |

- One of the two failures was a partial recovery: after a missed grasp the policy regrasped the block, but hit the
  edge of the bin and dropped it.
- **Short blocks are harder.** The training data has two block lengths, and this policy had some trouble with the
  shorter block. The sibling [ACT policy](https://huggingface.co/lichtphyz/act_v3_aug_desktop) succeeded 4/4 with short
  blocks in an extra, unrecorded test, and 9/10 in its own
  [10-episode evaluation](https://huggingface.co/datasets/lichtphyz/rollout_act_v3_aug_desktop); it also got the block in
  on the two placements this policy failed, though it bumped the bin edge on both.
- All 10 episodes (both camera streams, joint states and actions) are in
  [lichtphyz/rollout_diffusion_v3_desktop](https://huggingface.co/datasets/lichtphyz/rollout_diffusion_v3_desktop).

<a class="flex" href="https://huggingface.co/spaces/lerobot/visualize_dataset?path=lichtphyz/rollout_diffusion_v3_desktop">
<img class="block dark:hidden" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl.svg"/>
<img class="hidden dark:block" src="https://huggingface.co/datasets/huggingface/badges/resolve/main/visualize-this-dataset-xl-dark.svg"/>
</a>

---

## Citation

If you use this policy, please cite the method linked in the description above, along with LeRobot:

```bibtex
@misc{cadene2024lerobot,
    author = {Cadene, Remi and Alibert, Simon and Soare, Alexander and Gallouedec, Quentin and Zouitine, Adil and Palma, Steven and Kooijmans, Pepijn and Aractingi, Michel and Shukor, Mustafa and Aubakirova, Dana and Russi, Martino and Capuano, Francesco and Pascal, Caroline and Choghari, Jade and Moss, Jess and Wolf, Thomas},
    title = {LeRobot: State-of-the-art Machine Learning for Real-World Robotics in Pytorch},
    howpublished = "\url{https://github.com/huggingface/lerobot}",
    year = {2024}
}
```