Instructions to use Bigenlight/act_banana_in_pot_ee with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- LeRobot
How to use Bigenlight/act_banana_in_pot_ee with LeRobot:
- Notebooks
- Google Colab
- Kaggle
Add model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,60 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: cc-by-nc-4.0
|
| 3 |
+
library_name: lerobot
|
| 4 |
+
pipeline_tag: robotics
|
| 5 |
+
tags:
|
| 6 |
+
- robotics
|
| 7 |
+
- lerobot
|
| 8 |
+
- act
|
| 9 |
+
- imitation-learning
|
| 10 |
+
- ur7e
|
| 11 |
+
- end-effector
|
| 12 |
+
---
|
| 13 |
+
|
| 14 |
+
# ACT · banana-in-pot · EEF (10-D) — checkpoint 40k
|
| 15 |
+
|
| 16 |
+
Action Chunking Transformer (ACT) trained on the **end-effector (EEF) action space**
|
| 17 |
+
for the task *"put the right banana in the pot"* (UR7e arm, GELLO teleoperation,
|
| 18 |
+
LeRobot v3.0). This is the **40k-step checkpoint**, selected as best by open-loop MAE.
|
| 19 |
+
|
| 20 |
+
This is the EEF counterpart of the joint-space model
|
| 21 |
+
[`Bigenlight/act_banana_in_pot`](https://huggingface.co/Bigenlight/act_banana_in_pot).
|
| 22 |
+
|
| 23 |
+
## Action / observation space
|
| 24 |
+
- `observation.state` / `action`: **10-D** = `[x, y, z, r1..r6 (Zhou 6D rotation), gripper]`
|
| 25 |
+
— absolute next-frame TCP pose (xyz in metres) + gripper. (The joint model uses 7-D
|
| 26 |
+
`[q1..q6, gripper]`.)
|
| 27 |
+
- Cameras: `observation.images.cam1`, `observation.images.cam2` (RGB, resized 360×640).
|
| 28 |
+
- Backbone: ResNet18 + VAE, `chunk_size=100`, ~51.6M params. Normalization: MEAN_STD.
|
| 29 |
+
|
| 30 |
+
## Training
|
| 31 |
+
- Recipe identical to the joint baseline `train_act_valdiag.sh` except dataset + steps:
|
| 32 |
+
`--dataset.eval_split=0.117` (held-out episodes 45–50), batch 8, seed 1000, 50k steps.
|
| 33 |
+
- Dataset: `banana_in_pot_ee_action` (51 eps / 21,524 frames, 30 fps), built from the raw
|
| 34 |
+
[`Bigenlight/banana_in_pot_raw`](https://huggingface.co/datasets/Bigenlight/banana_in_pot_raw)
|
| 35 |
+
via recorded `tcp_pose` (no FK needed).
|
| 36 |
+
- Hardware: single RTX A4000, ~2h43m. No overfitting (held-out eval_loss monotone to 0.4594@50k).
|
| 37 |
+
|
| 38 |
+
## Held-out results (open-loop, eps 45–50)
|
| 39 |
+
| checkpoint | pose MAE (m + 6D) | gripper acc |
|
| 40 |
+
|---|---|---|
|
| 41 |
+
| **40k (this)** | **0.05564** | **0.914** |
|
| 42 |
+
| 50k | 0.05564 | 0.911 |
|
| 43 |
+
|
| 44 |
+
Selected by open-loop MAE (repo convention), not by eval_loss.
|
| 45 |
+
|
| 46 |
+
> ⚠️ Note: EEF pose MAE mixes metres (xyz) and unitless 6D-rotation and is **not**
|
| 47 |
+
> directly comparable to the joint model's radian MAE. See the comparison writeup.
|
| 48 |
+
|
| 49 |
+
## Usage
|
| 50 |
+
```python
|
| 51 |
+
from lerobot.policies.act.modeling_act import ACTPolicy
|
| 52 |
+
policy = ACTPolicy.from_pretrained("Bigenlight/act_banana_in_pot_ee")
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
## Links
|
| 56 |
+
- Experiments repo, full report, reproducibility (Docker): GitHub
|
| 57 |
+
[`Bigenlight/banana-in-pot-experiments`](https://github.com/Bigenlight/banana-in-pot-experiments)
|
| 58 |
+
— see `docs/ACT_EE_RESULTS.md` and `docs/JOINT_VS_EEF_ACT_COMPARISON.md`.
|
| 59 |
+
|
| 60 |
+
*License: CC-BY-NC-4.0 (trained on real-lab teleoperation video).*
|