README.md

---
license: apache-2.0
library_name: lerobot
tags:
  - act
  - robotics
  - lerobot
  - ur7e
  - imitation-learning
pipeline_tag: robotics
datasets:
  - Bigenlight/banana_in_pot_lerobot_v3
---

# ACT — Put the right banana in the pot (UR7e)

An **Action Chunking Transformer (ACT)** policy trained by imitation learning to perform
the manipulation task *"put the right banana in the pot"* on a **Universal Robots UR7e**
arm with two RGB cameras.

- **Policy:** ACT (Transformer encoder–decoder with a CVAE), **ResNet18** vision backbone
  (ImageNet-pretrained), `chunk_size = 100`, `n_action_steps = 100`.
- **Trained on:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3)
  (51 teleoperated episodes / 21,524 frames, UR7e follower + GELLO leader, 2 cameras).
- **Training length:** 50,000 steps, batch size 8, AdamW `lr = 1e-5` (constant).
- **Framework:** [LeRobot](https://github.com/huggingface/lerobot) v0.6.1.

## What the policy does

| I/O | Spec |
|---|---|
| `observation.state` | `(7,)` — UR joints `q1..q6` (radians) + gripper position |
| `observation.images.cam1` / `cam2` | RGB, trained at **360×640** |
| `action` | `(7,)` — `[cmd1..cmd6, grip_cmd]`, **absolute** joint targets (radians) + ~binary gripper |

Images were captured at 720p and **resized on-the-fly to 360×640** during training
(aspect-preserving half-resolution; no dataset re-encode). **You must resize every live
camera frame to 360×640** at inference — a different aspect ratio or interpolation degrades
the policy.

## Data & hardware setup

| Component | Detail |
|---|---|
| Robot | **Universal Robots UR7e** — 6-DOF collaborative arm, joints in radians. Inference uses the UR7e follower only. |
| Teleoperation (data collection) | **GELLO** low-cost 3D-printed leader arm. Leader signals are recorded but are **not** policy inputs at inference. |
| Camera 1 | **Intel RealSense D435** — RGB only |
| Camera 2 | **Intel RealSense D435if** — RGB only |
| Camera streams | 1280×720 (720p) @ **30 fps**, color only (**no depth / IR** recorded). ACT is trained on the two RGB views **resized to 360×640**. |
| Task | *"put the right banana in the pot"* — table with distractor objects (2 bananas, apple, carrots/peppers, watermelon slice) and a silver pot; success = the correct banana placed in the pot. |
| Dataset scale | 51 episodes / 21,524 frames / ~12 min @ 30 fps. |

## Training configuration

| Item | Value |
|---|---|
| Policy | ACT (ResNet18 backbone, CVAE, `chunk_size=100`, `n_action_steps=100`) |
| `dim_model` / heads | 512 / 8 |
| Encoder / decoder layers | 4 / 1 |
| Normalization | MEAN_STD for state, action, and visual |
| Batch / steps | 8 / 50,000 |
| Optimizer | AdamW, `lr = 1e-5` (constant), backbone `lr = 1e-5` |
| Backbone | `ResNet18_Weights.IMAGENET1K_V1` (ImageNet-pretrained) |
| GPU | RTX 3060 12GB (~4.7 GB used, ~3.9 step/s) |

### Training loss

![training loss](assets/loss_curve.png)

Final training loss ≈ **0.065**.

## Offline evaluation (open-loop: predicted actions vs. ground truth)

Evaluated on held-out episodes (train vs. held-out gap is negligible → generalizes, no
overfitting). The **step 50,000** checkpoint (this model) is the best by held-out L1.

| step | joints MAE (rad) | overall L1 | gripper acc | train L1 | held-out L1 |
|---|---|---|---|---|---|
| 10000 | 0.0392 | 0.0395 | 98.0% | 0.0392 | 0.0406 |
| 20000 | 0.0354 | 0.0345 | 98.8% | 0.0324 | 0.0356 |
| 30000 | 0.0267 | 0.0265 | 98.8% | 0.0258 | 0.0269 |
| 40000 | 0.0256 | 0.0245 | 99.2% | 0.0222 | 0.0256 |
| **50000** ⭐ | **0.0237** | **0.0225** | **99.2%** | 0.0205 | **0.0235** |

![evaluation trend](assets/eval_trend.png)

**Best checkpoint (step 50,000):**
- Held-out overall **L1 = 0.0235 rad** (≈ 1.34°); train L1 = 0.0205 → near-zero gap.
- Joints **MAE = 0.0237 rad** (≈ 1.36°).
- Gripper accuracy **99.2%**.

![per-joint MAE](assets/perjoint_best.png)

The wrist joint (`cmd6`) carries the largest error — it is the axis with the most natural
variation. The gripper channel is close to binary.

## How to use

### Load the policy (LeRobot)

```python
from lerobot.policies.act import ACTPolicy
from lerobot.policies import make_pre_post_processors

policy = ACTPolicy.from_pretrained("Bigenlight/act_banana_in_pot")
policy.eval()

# Normalization is NOT baked into forward() in lerobot 0.6.1 — it lives in the
# processor pipeline saved alongside the checkpoint. select_action returns a
# NORMALIZED action; the post-processor converts it back to radians.
preprocessor, postprocessor = make_pre_post_processors(
    policy_cfg=policy.config,
    pretrained_path="Bigenlight/act_banana_in_pot",
)
```

Per control tick:

```python
policy.reset()                       # once at the start of each rollout
obs = preprocessor(obs)              # normalize + batch + move to device
action = policy.select_action(obs)   # normalized
action = postprocessor(action)       # radians, numpy (7,)
```

`select_action` returns one action per call from an internal queue; on an empty queue it
predicts a full 100-step chunk and replans after 100 executed actions (temporal ensembling
is **off** by default in this config).

### Deploy on a real UR7e

LeRobot ships **no UR robot class** — you build the observation dict yourself and stream
joint targets with [`ur_rtde`](https://sdurobotics.gitlab.io/ur_rtde/) at **30 Hz**
(33.3 ms control period). Loop: read `getActualQ()` + gripper → grab both cameras, BGR→RGB,
resize to 360×640 → build `observation.state` / `observation.images.cam1` / `cam2` →
preprocess → `select_action` → postprocess → `servoJ(q_target[:6])` + drive gripper from
`grip_cmd`.

**⚠️ Safety — actions are ABSOLUTE joint positions (the single biggest safety driver):**

1. **Start near the dataset initial pose** `q1..q6 ≈ [2.84, -1.41, 1.78, -2.01, -1.66, -3.42]`
   rad before enabling the policy, or the first absolute command is a large jump.
2. **First-command jump guard:** if `max(|q_target − getActualQ()|) > ~0.15 rad`, **abort**.
3. **Clamp per-tick joint change** (e.g. ≤ 0.05–0.1 rad / 33 ms early on) and clamp to UR
   software joint limits.
4. **Reduced speed for first trials**; keep a hand on the **E-stop** dead-man switch.
5. **Camera mapping matters:** the policy learned a fixed `cam1`/`cam2` → physical-viewpoint
   mapping. Swap them and it fails silently. Verify wiring every session.
6. **Gripper** `grip_cmd` is ~binary — threshold (e.g. `>0.5 → close`) and map to your driver.

A full deployment guide (with a reference `ur_rtde` skeleton and all the citations to the
lerobot inference/normalization code paths) is in the project's `DEPLOY_UR.md`.

## Related repositories

- **Dataset:** [`Bigenlight/banana_in_pot_lerobot_v3`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_lerobot_v3)
- **HIL-SERL prep bundle** (reward classifier, SAC config, runbook) for taking this task
  online with reinforcement learning:
  [`Bigenlight/banana_in_pot_hilserl`](https://huggingface.co/Bigenlight/banana_in_pot_hilserl)

## Limitations

- **Small dataset** (51 demos). Metrics reflect an easy, temporally-correlated task; expect
  reduced robustness to novel object positions, lighting, or camera placement.
- **Offline metrics only.** All numbers above are open-loop (predicted vs. ground-truth
  actions). Real closed-loop task success on the arm has not been measured here — a physical
  UR7e deployment is required for the final verdict.
- **Absolute-joint action space** demands the safety guards above; the policy was only ever
  conditioned on states near the data-collection start pose.
- The ResNet18 backbone is **ImageNet-pretrained** (not robotics-pretrained); the CVAE and
  transformer are trained from scratch on this task.