Upload 4 files
Browse files- 1. LeRobot Installation.md +82 -0
- 2. LeRobot Setting.md +187 -0
- 3. LeRobot Manipulation.md +211 -0
- 4. LeRobot VLA.md +246 -0
1. LeRobot Installation.md
ADDED
|
@@ -0,0 +1,82 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# LeRobot Setting Guide
|
| 2 |
+
https://huggingface.co/docs/lerobot/installation \
|
| 3 |
+
https://huggingface.co/docs/lerobot/so101
|
| 4 |
+
|
| 5 |
+
last updated Dec/26/2025
|
| 6 |
+
|
| 7 |
+
# 1. LeRobot Installation
|
| 8 |
+
|
| 9 |
+
## 1-1. Python ๊ฐ์ ํ๊ฒฝ ์์ฑ
|
| 10 |
+
```bash
|
| 11 |
+
# Create python 3.10 conda env
|
| 12 |
+
conda create -n lerobot python=3.10 -y
|
| 13 |
+
|
| 14 |
+
# Activate env
|
| 15 |
+
conda activate lerobot
|
| 16 |
+
```
|
| 17 |
+
|
| 18 |
+
---
|
| 19 |
+
|
| 20 |
+
## 1-2. ffmpeg, Rerun ์ค์น
|
| 21 |
+
- ์์ ์ฒ๋ฆฌ๋ฅผ ์ํด `ffmpeg` ํ์
|
| 22 |
+
- LeRobot ๋ฐ์ดํฐ visualization์ ์ํด `Rerun` ํ์
|
| 23 |
+
```bash
|
| 24 |
+
conda install ffmpeg=7.1.1 -c conda-forge
|
| 25 |
+
pip install rerun-sdk
|
| 26 |
+
```
|
| 27 |
+
|
| 28 |
+
---
|
| 29 |
+
|
| 30 |
+
## 1-3. LeRobot ์์ค ์ฝ๋ ๋ค์ด๋ก๋
|
| 31 |
+
```bash
|
| 32 |
+
# LeRobot github repo clone
|
| 33 |
+
git clone https://github.com/huggingface/lerobot.git
|
| 34 |
+
cd lerobot
|
| 35 |
+
```
|
| 36 |
+
|
| 37 |
+
---
|
| 38 |
+
|
| 39 |
+
## 1-4. LeRobot ์ค์น
|
| 40 |
+
```bash
|
| 41 |
+
pip install -e .
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
---
|
| 45 |
+
|
| 46 |
+
## 1-5. ๋ก๋ด ํ๋์จ์ด ๋๋ผ์ด๋ฒ
|
| 47 |
+
์ฌ์ฉํ๋ ๋ชจํฐ์ ๋ง๋ ๋๋ผ์ด๋ฒ ์ค์น
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
# Feetech motor (SO-100, SO-101)
|
| 51 |
+
pip install -e ".[feetech]"
|
| 52 |
+
|
| 53 |
+
# Dynamixel motor (Koch, etc.)
|
| 54 |
+
pip install -e ".[dynamixel]"
|
| 55 |
+
```
|
| 56 |
+
|
| 57 |
+
---
|
| 58 |
+
|
| 59 |
+
## 1-6. ์ค์น ํ์ธ
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
python -c "import lerobot; print(lerobot.__version__)"
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
---
|
| 66 |
+
|
| 67 |
+
## (Optional) ์๋ฎฌ๋ ์ด์
ํ๊ฒฝ
|
| 68 |
+
ํน์ ์๋ฎฌ๋ ์ด์
ํ๊ฒฝ์ ์ฌ์ฉํ๋ ค๋ฉด ์ถ๊ฐ ํจํค์ง ์ค์น
|
| 69 |
+
|
| 70 |
+
```bash
|
| 71 |
+
# ALOHA ์๋ฎฌ๋ ์ด์
ํ๊ฒฝ
|
| 72 |
+
pip install -e ".[aloha]"
|
| 73 |
+
|
| 74 |
+
# Push-T ์๋ฎฌ๋ ์ด์
ํ๊ฒฝ
|
| 75 |
+
pip install -e ".[pusht]"
|
| 76 |
+
|
| 77 |
+
# ๋ชจ๋ ์๋ฎฌ๋ ์ด์
ํ๊ฒฝ
|
| 78 |
+
pip install -e ".[aloha,pusht,xarm]"
|
| 79 |
+
```
|
| 80 |
+
|
| 81 |
+
---
|
| 82 |
+
|
2. LeRobot Setting.md
ADDED
|
@@ -0,0 +1,187 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 2. LeRobot Setting
|
| 2 |
+
last updated Dec/26/2025
|
| 3 |
+
|
| 4 |
+
## 2-1. USB port ์ฐพ๊ธฐ
|
| 5 |
+
|
| 6 |
+
1) Follower arm ๋ณด๋์ PC๋ฅผ USB๋ก ์ฐ๊ฒฐ
|
| 7 |
+
2) ์๋ ๋ช
๋ น ์คํ
|
| 8 |
+
|
| 9 |
+
```bash
|
| 10 |
+
lerobot-find-port
|
| 11 |
+
```
|
| 12 |
+
|
| 13 |
+
3) ์๋ด์ ๋ฐ๋ผ **Follower arm ๋ณด๋์์ USB ์ผ์ด๋ธ์ ๋ฝ๊ณ Enter**๋ฅผ ๋๋ฅด๋ฉด, ์๋์ ๊ฐ์ ๊ฒฐ๊ณผ๊ฐ ์ถ๋ ฅ๋ฉ๋๋ค.
|
| 14 |
+
|
| 15 |
+
```text
|
| 16 |
+
The port of this MotorsBus is '/dev/ttyACM0'
|
| 17 |
+
Reconnect the USB cable
|
| 18 |
+
```
|
| 19 |
+
|
| 20 |
+
โ ์ด ๊ฒฝ์ฐ์ follower arm ๋ณด๋์ USB ํฌํธ๋ **`/dev/ttyACM0`** ์
๋๋ค.
|
| 21 |
+
|
| 22 |
+
4) Leader arm ๋ณด๋๋ ๋ง์ฐฌ๊ฐ์ง๋ก USB ํฌํธ๋ฅผ ์ฐพ์์ค๋๋ค.
|
| 23 |
+
|
| 24 |
+
5) ๊ฐ ํฌํธ๊ฐ ์ด๋ค arm(Follower/Leader)์ ํด๋นํ๋์ง **๋ฐ๋์ ๊ตฌ๋ถ**ํด ๋์ธ์.
|
| 25 |
+
|
| 26 |
+
6) ์ฐ๊ฒฐ๋ USB ํฌํธ์ Read/Write ๊ถํ ๋ถ์ฌ
|
| 27 |
+
```bash
|
| 28 |
+
sudo chmod 666 /dev/ttyACM0
|
| 29 |
+
sudo chmod 666 /dev/ttyACM1
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
## (Optional) Serial number๋ฅผ ๊ธฐ๋ฐ์ผ๋ก USB port ๊ณ ์ ํ๋ ๋ฒ
|
| 33 |
+
|
| 34 |
+
### 1. ttyACM0 serial ํ์ธ
|
| 35 |
+
```bash
|
| 36 |
+
udevadm info -a -n /dev/ttyACM0 | grep '{serial}' -m 1
|
| 37 |
+
```
|
| 38 |
+
|
| 39 |
+
```bash
|
| 40 |
+
# Example print
|
| 41 |
+
ATTRS{serial}=="5A7A057786"
|
| 42 |
+
```
|
| 43 |
+
|
| 44 |
+
### 2. ttyACM1 serial ํ์ธ
|
| 45 |
+
```bash
|
| 46 |
+
udevadm info -a -n /dev/ttyACM1 | grep '{serial}' -m 1
|
| 47 |
+
```
|
| 48 |
+
|
| 49 |
+
```bash
|
| 50 |
+
# Example print
|
| 51 |
+
ATTRS{serial}=="5A68011549"
|
| 52 |
+
```
|
| 53 |
+
|
| 54 |
+
### 3. udev rule ์์ฑ
|
| 55 |
+
|
| 56 |
+
```bash
|
| 57 |
+
# udev rule ์์ฑ/ํธ์ง
|
| 58 |
+
sudo nano /etc/udev/rules.d/99-serial.rules
|
| 59 |
+
```
|
| 60 |
+
|
| 61 |
+
```bash
|
| 62 |
+
GNU nano 2.9.3 /etc/udev/rules.d/99-serial.rules
|
| 63 |
+
|
| 64 |
+
# ์๋์ serial number๋ ์์์
๋๋ค. ์์ ์ ํ๊ฒฝ์์ ํ์ธํ ๊ฐ์ผ๋ก ๋ฐ๋์ ๊ต์ฒดํ์ธ์.
|
| 65 |
+
SUBSYSTEM=="tty", ATTRS{serial}=="5A7A057786", SYMLINK+="so101_follower"
|
| 66 |
+
SUBSYSTEM=="tty", ATTRS{serial}=="5A68011549", SYMLINK+="so101_leader"
|
| 67 |
+
```
|
| 68 |
+
|
| 69 |
+
### 4. rule reload / trigger
|
| 70 |
+
```bash
|
| 71 |
+
sudo udevadm control --reload-rules
|
| 72 |
+
sudo udevadm trigger
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
### 5. ๊ณ ์ ํฌํธ ํ์ธ
|
| 76 |
+
```bash
|
| 77 |
+
ls -l /dev/so101_*
|
| 78 |
+
```
|
| 79 |
+
|
| 80 |
+
```bash
|
| 81 |
+
# Example print
|
| 82 |
+
# USB ํฌํธ๋ฅผ ๊ฝ๋ ์์๊ฐ ๋ฐ๋์ด๋ ๋ค์ ๊ฒฝ๋ก๋ก ํญ์ ๋์ผํ๊ฒ ์ธ์
|
| 83 |
+
lrwxrwxrwx 1 root root 7 11์ 24 21:46 /dev/so101_follower -> ttyACM0
|
| 84 |
+
lrwxrwxrwx 1 root root 7 11์ 24 21:46 /dev/so101_leader -> ttyACM1
|
| 85 |
+
```
|
| 86 |
+
|
| 87 |
+
---
|
| 88 |
+
|
| 89 |
+
## 2-2. ๋ชจํฐ ID ์ค์
|
| 90 |
+
|
| 91 |
+
- ๊ฐ ๋ชจํฐ์ **๊ณ ์ ID** ์ค์ ์ ์ํด ์๋ ๋ช
๋ น์ด ์คํ ํ, ํฐ๋ฏธ๋์ ํ์๋๋ ๊ฐ์ด๋์ ๋ฐ๋ผ **๋ชจํฐ๋ฅผ ์์๋๋ก ์ฐ๊ฒฐ**ํ๋ฉฐ ์งํ
|
| 92 |
+
- ์๋ ๋งํฌ์ ์์์ ๋ณด๋ฉด์ ํ๋ ๊ฑธ ์ถ์ฒํฉ๋๋ค. \
|
| 93 |
+
https://huggingface.co/docs/lerobot/so101#2-set-the-motors-ids-and-baudrates
|
| 94 |
+
|
| 95 |
+
### Follower
|
| 96 |
+
```bash
|
| 97 |
+
lerobot-setup-motors \
|
| 98 |
+
--robot.type=so101_follower \
|
| 99 |
+
--robot.port=/dev/so101_follower # Follower ๋ณด๋์ USB port
|
| 100 |
+
```
|
| 101 |
+
|
| 102 |
+
### Leader
|
| 103 |
+
```bash
|
| 104 |
+
lerobot-setup-motors \
|
| 105 |
+
--teleop.type=so101_leader \
|
| 106 |
+
--teleop.port=/dev/so101_leader # Leader ๋ณด๋์ USB port
|
| 107 |
+
```
|
| 108 |
+
|
| 109 |
+
---
|
| 110 |
+
|
| 111 |
+
## 2-3. Calibration
|
| 112 |
+
|
| 113 |
+
- Calibration์ ๋ก๋ด ํ์ ๊ฐ ๊ด์ (joint)์ด ์ ํํ ์์น๋ฅผ ์ธ์ํ๊ณ ๋งคํํ ์ ์๋๋ก ํ๋ ์ค์ํ ๊ณผ์ ์
๋๋ค.
|
| 114 |
+
- ์๋ ๋งํฌ์ ์์์ ๋ณด๋ฉด์ ํ๋ ๊ฑธ ์ถ์ฒํฉ๋๋ค. \
|
| 115 |
+
https://huggingface.co/docs/lerobot/so101#calibrate
|
| 116 |
+
|
| 117 |
+
### Follower
|
| 118 |
+
```bash
|
| 119 |
+
lerobot-calibrate \
|
| 120 |
+
--robot.type=so101_follower \
|
| 121 |
+
--robot.port=/dev/so101_follower \
|
| 122 |
+
--robot.id=follower
|
| 123 |
+
```
|
| 124 |
+
|
| 125 |
+
### Leader
|
| 126 |
+
```bash
|
| 127 |
+
lerobot-calibrate \
|
| 128 |
+
--teleop.type=so101_leader \
|
| 129 |
+
--teleop.port=/dev/so101_leader \
|
| 130 |
+
--teleop.id=leader
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
### Calibration ๊ณผ์
|
| 134 |
+
Command ์คํ ํ ๋ค์๊ณผ ๊ฐ์ ๋ฉ์์ง๊ฐ ํ์๋ฉ๋๋ค.
|
| 135 |
+
|
| 136 |
+
```text
|
| 137 |
+
Move None SO101Leader to the middle of its range of motion and press ENTER....
|
| 138 |
+
Move all joints sequentially through their entire ranges of motion.
|
| 139 |
+
Recording positions. Press ENTER to stop...
|
| 140 |
+
```
|
| 141 |
+
|
| 142 |
+
์ ๋ฉ์์ง๋ฅผ ๋ฐ๋ผ **์ค๊ฐ ์์น(middle position)** ๋ฅผ ์ก์์ฃผ๊ณ **Enter**๋ฅผ ๋๋ฌ์ฃผ์ธ์.
|
| 143 |
+
|
| 144 |
+
#### Joint ์์ง์ ์ฒดํฌ๋ฆฌ์คํธ
|
| 145 |
+
- ๋ก๋ด ํ์ **์ค๊ฐ ์์น(middle position)** ๋ก ์ด๋
|
| 146 |
+
- ๋ชจ๋ ๊ด์ (joints)์ **์์ฐจ์ ์ผ๋ก** ์ ์ฒด ๋ฒ์์์ ์์ง์ฌ๋ณด๊ธฐ
|
| 147 |
+
- **ENTER** ํค๋ฅผ ๋๋ฌ ๊ธฐ๋ก ์ค์ง
|
| 148 |
+
|
| 149 |
+
### Calibration ๊ฒฐ๊ณผ
|
| 150 |
+
Calibration์ด ์๋ฃ๋๋ฉด ๋ค์๊ณผ ๊ฐ์ ๊ด์ ์์น ์ ๋ณด๊ฐ ์ถ๋ ฅ๋ฉ๋๋ค(์์).
|
| 151 |
+
|
| 152 |
+
```bash
|
| 153 |
+
-------------------------------------------
|
| 154 |
+
-------------------------------------------
|
| 155 |
+
NAME | MIN | POS | MAX
|
| 156 |
+
shoulder_pan | 1955 | 2007 | 3132
|
| 157 |
+
shoulder_lift | 709 | 724 | 2057
|
| 158 |
+
elbow_flex | 1605 | 3036 | 3110
|
| 159 |
+
wrist_flex | 808 | 2797 | 2803
|
| 160 |
+
wrist_roll | 2037 | 2048 | 2055
|
| 161 |
+
gripper | 2019 | 2046 | 3383
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
### ํ์ผ ์ ์ฅ ํ์ธ
|
| 165 |
+
์ฑ๊ณต์ ์ผ๋ก ์๋ฃ๋๋ฉด ๋ค์ ๋ฉ์์ง๊ฐ ํ์๋ฉ๋๋ค.
|
| 166 |
+
|
| 167 |
+
```bash
|
| 168 |
+
Calibration saved to ~/.cache/huggingface/lerobot/calibration/teleoperators/so101_leader/leader.json
|
| 169 |
+
```
|
| 170 |
+
|
| 171 |
+
### Calibration ํ์ผ
|
| 172 |
+
|
| 173 |
+
Calibration์ด ์๋ฃ๋๋ฉด ๋ค์ ์์น์ ํ์ผ์ด ์ ์ฅ๋ฉ๋๋ค.
|
| 174 |
+
|
| 175 |
+
- ๋ฆฌ๋ ์:
|
| 176 |
+
`~/.cache/huggingface/lerobot/calibration/teleoperators/so101_leader/leader.json`
|
| 177 |
+
- ํ๋ก์ ์:
|
| 178 |
+
`~/.cache/huggingface/lerobot/calibration/robots/so101_follower/follower.json`
|
| 179 |
+
|
| 180 |
+
|
| 181 |
+
๊ฐ Calibration ํ์ผ์๋ ๋ค์ ์ ๏ฟฝ๏ฟฝ๊ฐ ํฌํจ๋ฉ๋๋ค.
|
| 182 |
+
|
| 183 |
+
- ๊ฐ ๊ด์ ์ ์ต์๊ฐ(MIN)
|
| 184 |
+
- ํ์ฌ ์์น(POS)
|
| 185 |
+
- ์ต๋๊ฐ(MAX)
|
| 186 |
+
|
| 187 |
+
---
|
3. LeRobot Manipulation.md
ADDED
|
@@ -0,0 +1,211 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 3. LeRobot Manipulation
|
| 2 |
+
last updated Dec/26/2025
|
| 3 |
+
|
| 4 |
+
## 3-1. Teleoperation
|
| 5 |
+
|
| 6 |
+
### ์นด๋ฉ๋ผ index, ์คํ, fps ํ์ธ
|
| 7 |
+
2๊ฐ ์ด์์ ์นด๋ฉ๋ผ๋ฅผ ์ฐ๊ฒฐํด ๋ฐ์ดํฐ์
๊ตฌ์ถ/๋ชจ๋ธ ์ถ๋ก ์ ํ์ฉํ ์ ์์ต๋๋ค.
|
| 8 |
+
PC์ ์ฐ๊ฒฐ๋ ์นด๋ฉ๋ผ์ index, ํด์๋, fps ๋ฑ ์ ๋ณด๋ฅผ ํ์ธํ๋ ค๋ฉด ์๋ ๋ช
๋ น์ด๋ฅผ ์คํํ์ธ์.
|
| 9 |
+
|
| 10 |
+
```bash
|
| 11 |
+
lerobot-find-cameras
|
| 12 |
+
```
|
| 13 |
+
|
| 14 |
+
```bash
|
| 15 |
+
# Example print
|
| 16 |
+
--- Detected Cameras ---
|
| 17 |
+
Camera #0:
|
| 18 |
+
Name: OpenCV Camera @ /dev/video0
|
| 19 |
+
Type: OpenCV
|
| 20 |
+
Id: /dev/video0
|
| 21 |
+
Backend api: V4L2
|
| 22 |
+
Default stream profile:
|
| 23 |
+
Format: 0.0
|
| 24 |
+
Fourcc: YUYV
|
| 25 |
+
Width: 640
|
| 26 |
+
Height: 480
|
| 27 |
+
Fps: 30.0
|
| 28 |
+
--------------------
|
| 29 |
+
Camera #1:
|
| 30 |
+
Name: OpenCV Camera @ /dev/video2
|
| 31 |
+
Type: OpenCV
|
| 32 |
+
Id: /dev/video2
|
| 33 |
+
Backend api: V4L2
|
| 34 |
+
Default stream profile:
|
| 35 |
+
Format: 0.0
|
| 36 |
+
Fourcc: YUYV
|
| 37 |
+
Width: 640
|
| 38 |
+
Height: 480
|
| 39 |
+
Fps: 30.0
|
| 40 |
+
--------------------
|
| 41 |
+
|
| 42 |
+
Finalizing image saving...
|
| 43 |
+
Image capture finished. Images saved to outputs/captured_images
|
| 44 |
+
```
|
| 45 |
+
|
| 46 |
+
> **TIP**: ์นด๋ฉ๋ผ ์ฐ๊ฒฐ ์์์ ๋ฐ๋ผ index๊ฐ ๋ฌ๋ผ์ง ์ ์์ผ๋, ์คํ ์ ํ๋ก ํญ์ `lerobot-find-cameras` ๋ค์์ `.../lerobot/outputs/captured_images`์์ ์บก์ฒ๋ ์ฌ์ง์ ํตํด ๊ฐ camera์ ํฌํธ๋ฅผ ํ์ธํ์ธ์.
|
| 47 |
+
|
| 48 |
+
### Teleoperation command
|
| 49 |
+
```bash
|
| 50 |
+
lerobot-teleoperate \
|
| 51 |
+
--robot.type=so101_follower \
|
| 52 |
+
--robot.port=/dev/so101_follower \
|
| 53 |
+
--robot.id=follower_new_gripper \
|
| 54 |
+
--robot.cameras="{
|
| 55 |
+
top: {type: opencv, index_or_path: "/dev/video0", width: 640, height: 480, fps: 30},
|
| 56 |
+
wrist: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}
|
| 57 |
+
}" \
|
| 58 |
+
--teleop.type=so101_leader \
|
| 59 |
+
--teleop.port=/dev/so101_leader \
|
| 60 |
+
--teleop.id=leader \
|
| 61 |
+
--display_data=true
|
| 62 |
+
```
|
| 63 |
+
|
| 64 |
+
### ์ต์
์ค๋ช
|
| 65 |
+
| ์ต์
๋ช
| ์ค๋ช
| ์์ ๊ฐ |
|
| 66 |
+
|---|---|---|
|
| 67 |
+
| `--robot.type` | follower ๋ก๋ด ํ์
| `so101_follower` |
|
| 68 |
+
| `--robot.port` | follower USB ํฌํธ | `/dev/so101_follower` |
|
| 69 |
+
| `--robot.id` | follower ๋ก๋ด ๊ณ ์ ID (calibration ๋ ์ค์ ํ ID) |
|
| 70 |
+
| `--robot.cameras` | ์นด๋ฉ๋ผ ์ ๋ณด | `lerobot-find-cameras`๋ก ํ์ธํ ๊ฐ |
|
| 71 |
+
| `--teleop.type` | leader ๋ก๋ด ํ์
| `so101_leader` |
|
| 72 |
+
| `--teleop.port` | leader USB ํฌํธ | `/dev/so101_leader` |
|
| 73 |
+
| `--teleop.id` | leader ์ด๋ฆ | `leader` |
|
| 74 |
+
|
| 75 |
+
---
|
| 76 |
+
|
| 77 |
+
## 3-2. Record
|
| 78 |
+
|
| 79 |
+
### HuggingFace ๊ณ์ ์ค์
|
| 80 |
+
|
| 81 |
+
```bash
|
| 82 |
+
# HuggingFace CLI ํ ํฐ ๋ก๊ทธ์ธ
|
| 83 |
+
hf auth login --token ${YOUR_TOKEN_HERE} --add-to-git-credential
|
| 84 |
+
|
| 85 |
+
# ๋ก๊ทธ์ธ ํ์ธ
|
| 86 |
+
HF_USER=$(hf auth whoami | head -n 1)
|
| 87 |
+
export HF_USER=$(python -c "from huggingface_hub import whoami; print(whoami().get(\"name\", \"\"))")
|
| 88 |
+
|
| 89 |
+
# (Optional) ์๊ตฌ ํ๊ฒฝ ๋ณ์ ์ค์
|
| 90 |
+
echo 'export HF_USER=$(python -c "from huggingface_hub import whoami; print(whoami().get(\"name\", \"\"))")' >> ~/.bashrc
|
| 91 |
+
source ~/.bashrc
|
| 92 |
+
echo ${HF_USER}
|
| 93 |
+
|
| 94 |
+
# Task Name ํ๊ฒฝ ๋ณ์ ์ค์
|
| 95 |
+
echo export ${TASK_NAME}=TASK_NAME >> ~/.bashrc # Task ๋ฐ๋ ๋๋ง๋ค ์๋ก ์ค์
|
| 96 |
+
|
| 97 |
+
#-------------------------------------------------------------------
|
| 98 |
+
# Huggingface ๋ก๊ทธ์ธ ํ์ง ์๊ณ ๋ก์ปฌ์์ ์งํํ๋ ๊ฒฝ์ฐ
|
| 99 |
+
echo export ${HF_USER}=YOUR_NAME >> ~/.bashrc
|
| 100 |
+
echo export ${TASK_NAME}=TASK_NAME >> ~/.bashrc
|
| 101 |
+
source ~/.bashrc
|
| 102 |
+
echo ${HF_USER} && echo ${TASK_NAME}
|
| 103 |
+
```
|
| 104 |
+
|
| 105 |
+
- HuggingFace [ํ ํฐ ์ค์ ํ์ด์ง](https://huggingface.co/settings/tokens)์์ ์ ํ ํฐ ์์ฑ
|
| 106 |
+
- ๊ถํ: `Write` ์ ํ (๋์ค์ pretrained model์ ํ์ต์ํค๋ ๊ฒฝ์ฐ `Read` ๊ถํ๋ ํ์ํ๋ฏ๋ก `Fine-grained` ์ ํ ์ถ์ฒ)
|
| 107 |
+
- ์์ฑ๋ ํ ํฐ์ ์ ๋ช
๋ น์ด์ `YOUR_TOKEN_HERE`์ ์
๋ ฅ
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
### Record command
|
| 111 |
+
```bash
|
| 112 |
+
lerobot-record \
|
| 113 |
+
--robot.type=so101_follower \
|
| 114 |
+
--robot.port=/dev/ttyACM0 \
|
| 115 |
+
--robot.id=follower_new_gripper \
|
| 116 |
+
--robot.cameras="{
|
| 117 |
+
top: {type: opencv, index_or_path: "/dev/video0", width: 640, height: 480, fps: 30},
|
| 118 |
+
wrist: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}
|
| 119 |
+
}" \
|
| 120 |
+
--teleop.type=so101_leader \
|
| 121 |
+
--teleop.port=/dev/ttyACM1 \
|
| 122 |
+
--teleop.id=leader \
|
| 123 |
+
--dataset.single_task="make a cup of coffee." \
|
| 124 |
+
--dataset.repo_id=${HF_USER}/${TASK_NAME} \
|
| 125 |
+
--dataset.num_episodes=10 \
|
| 126 |
+
--dataset.episode_time_s=1200 \
|
| 127 |
+
--dataset.reset_time_s=3 \
|
| 128 |
+
--display_data=true \
|
| 129 |
+
--dataset.push_to_hub=true \
|
| 130 |
+
--resume=true # ์ด๋ฏธ ์์ฑ๋ ๋ฐ์ดํฐ์
repo์ ์ถ๊ฐ๋ก ๋ฐ์ดํฐ ์์งํ๋ ๊ฒฝ์ฐ
|
| 131 |
+
```
|
| 132 |
+
|
| 133 |
+
### ์ฃผ์ Record ์ต์
|
| 134 |
+
|
| 135 |
+
| ์ต์
| ์ค๋ช
|
|
| 136 |
+
|------|------|
|
| 137 |
+
| `--robot.type` | follower ๋ก๋ด ํ์
|
|
| 138 |
+
| `--robot.port` | follower USB ํฌํธ |
|
| 139 |
+
| `--robot.id` | follower ๋ก๋ด ๊ณ ์ ID (calibration ๋ ์ค์ ํ ID) |
|
| 140 |
+
| `--robot.cameras` | ์นด๋ฉ๋ผ ์ ๋ณด |
|
| 141 |
+
| `--teleop.type` | leader ๋ก๋ด ํ์
|
|
| 142 |
+
| `--robot.port` | leader USB ํฌํธ |
|
| 143 |
+
| `--teleop.id` | leader ๋ก๋ด ๊ณ ์ ID (calibration ๋ ์ค์ ํ ID) |
|
| 144 |
+
| `--dataset.single_task` | language instruction |
|
| 145 |
+
| `--dataset.repo_id` | ๋ฐ์ดํฐ์
repo ์ด๋ฆ |
|
| 146 |
+
| `--dataset.num_episodes` | ์์งํ ์ํผ์๋ ์ |
|
| 147 |
+
| `--dataset.episode_time_s` | ๊ฐ ์ํผ์๋ ๋
นํ ์๊ฐ |
|
| 148 |
+
| `--dataset.reset_time_s` | ์ํผ์๋ ๏ฟฝ๏ฟฝ ๋ฆฌ์
์๊ฐ |
|
| 149 |
+
| `--display_data` | ์ค์๊ฐ ๋ฐ์ดํฐ ์๊ฐํ ์ฌ๋ถ (`Rerun`) |
|
| 150 |
+
| `--dataset.push_to_hub` | Huggingface ์๋ ์
๋ก๋ ์ฌ๋ถ |
|
| 151 |
+
|
| 152 |
+
|
| 153 |
+
### Record ์ ํค๋ณด๋ ๋จ์ถํค
|
| 154 |
+
| ํค | ๊ธฐ๋ฅ |
|
| 155 |
+
|----|------|
|
| 156 |
+
| โ | ์ํผ์๋ ์กฐ๊ธฐ ์ข
๋ฃ |
|
| 157 |
+
| โ | ์ํผ์๋ ์ฌ๊ธฐ๋ก |
|
| 158 |
+
| ESC | Command ์ข
๋ฃ ๋ฐ ์
๋ก๋ |
|
| 159 |
+
|
| 160 |
+
|
| 161 |
+
### FileExistsError
|
| 162 |
+
๋ฐ์ดํฐ์
repo๊ฐ ์ด๋ฏธ ์กด์ฌํ๋ฉด ๊ธฐ์กด ๋๋ ํ ๋ฆฌ๋ฅผ ๋ฎ์ด์ฐ์ง ์๊ณ FileExistsError๊ฐ ๋ฐ์ํ๋ฏ๋ก ๊ธฐ์กด repo๋ฅผ ์ญ์ ํ๊ฑฐ๋ `--resume=true` ์ต์
์ผ๋ก ๊ธฐ์กด repo์ ์ด์ด์ ์์งํด์ผ ํฉ๋๋ค.
|
| 163 |
+
|
| 164 |
+
```bash
|
| 165 |
+
FileExistsError: [Errno 17] File exists: '/home/.../so101_pickplace'
|
| 166 |
+
```
|
| 167 |
+
|
| 168 |
+
**ํด๊ฒฐ ๋ฐฉ๋ฒ:**
|
| 169 |
+
```bash
|
| 170 |
+
rm -rf ~/.cache/huggingface/lerobot/${HF_USER}/${TASK_NAME}
|
| 171 |
+
```
|
| 172 |
+
|
| 173 |
+
---
|
| 174 |
+
|
| 175 |
+
## 3-3. Replay
|
| 176 |
+
|
| 177 |
+
```bash
|
| 178 |
+
lerobot-replay \
|
| 179 |
+
--robot.type=so101_follower \
|
| 180 |
+
--robot.port=/dev/so101_follower \
|
| 181 |
+
--robot.id=follower \
|
| 182 |
+
--dataset.repo_id=${HF_USER}/${TASK_NAME} \
|
| 183 |
+
--dataset.episode=0
|
| 184 |
+
```
|
| 185 |
+
|
| 186 |
+
### ์ฃผ์ ์ต์
|
| 187 |
+
| ์ต์
| ์ค๋ช
|
|
| 188 |
+
|------|------|
|
| 189 |
+
| `--dataset.repo_id` | ์ฌ์ํ ๋ฐ์ดํฐ์
repo ์ด๋ฆ |
|
| 190 |
+
| `--dataset.episode` | ์ฌ์ํ ์ํผ์๋ ๋ฒํธ |
|
| 191 |
+
| `--dataset.fps` | ์ฌ์ ์๋ |
|
| 192 |
+
|
| 193 |
+
---
|
| 194 |
+
|
| 195 |
+
## 3-4. Visualization (๋ฐ์ดํฐ์
ํ์ธ)
|
| 196 |
+
|
| 197 |
+
### ๋ฐฉ๋ฒ 1: HuggingFace ์น์์ ํ์ธ
|
| 198 |
+
[LeRobot Dataset Visualizer](https://huggingface.co/spaces/lerobot/visualize_dataset) ํ์ด์ง์์ ๋ฐ์ดํฐ์
repo๋ฅผ ์
๋ ฅ ํ ํ์ธ
|
| 199 |
+
|
| 200 |
+
### ๋ฐฉ๋ฒ 2: ๋ก์ปฌ ์๊ฐํ
|
| 201 |
+
|
| 202 |
+
```bash
|
| 203 |
+
lerobot-dataset-viz \
|
| 204 |
+
--repo-id=${HF_USER}/${TASK_NAME} \
|
| 205 |
+
--episode-index=0 # visualizeํ ์ํผ์๋ ๋ฒํธ
|
| 206 |
+
```
|
| 207 |
+
- Rerun ๋ทฐ์ด ์๋ ์คํ
|
| 208 |
+
- ์นด๋ฉ๋ผ ์์, ๋ก๋ด ์ก์
, ์ํ ์ ๋ณด ํ์ธ ๊ฐ๋ฅ
|
| 209 |
+
|
| 210 |
+
---
|
| 211 |
+
|
4. LeRobot VLA.md
ADDED
|
@@ -0,0 +1,246 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# 4. LeRobot VLA
|
| 2 |
+
last updated Dec/26/2025
|
| 3 |
+
|
| 4 |
+
## 4-1. Train a policy
|
| 5 |
+
|
| 6 |
+
### Example train command (pi0.5)
|
| 7 |
+
|
| 8 |
+
#### (A) Multi-GPU (Accelerate)
|
| 9 |
+
- ๋ฉํฐ GPU๋ฅผ ์ฌ์ฉํ ๊ฒฝ์ฐ ์ผ๋ฐ์ ์ผ๋ก `CUDA_VISIBLE_DEVICES`์ ๋
ธ์ถํ **GPU ๊ฐ์ = `--num_processes`** ๋ก ๋ง์ถฅ๋๋ค.
|
| 10 |
+
|
| 11 |
+
```bash
|
| 12 |
+
# Multi-GPU example (4 GPUs)
|
| 13 |
+
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
| 14 |
+
accelerate launch --num_processes=4 $(which lerobot-train) \
|
| 15 |
+
--wandb.enable=true \
|
| 16 |
+
--wandb.project=lerobot \
|
| 17 |
+
--dataset.repo_id=${HF_USER}/${TASK_NAME} \
|
| 18 |
+
--output_dir=${HF_USER}/pi05_pickplace \
|
| 19 |
+
--policy.type=pi05 \
|
| 20 |
+
--policy.pretrained_path=lerobot/pi05_base \
|
| 21 |
+
--policy.dtype=bfloat16 \
|
| 22 |
+
--policy.device=cuda \
|
| 23 |
+
--steps=3000 \
|
| 24 |
+
--batch_size=64 \
|
| 25 |
+
--log_freq=100 \
|
| 26 |
+
--save_checkpoint=true \
|
| 27 |
+
--save_freq=500 \
|
| 28 |
+
--policy.push_to_hub=true \
|
| 29 |
+
--policy.repo_id=${HF_USER}/pi05_pickplace
|
| 30 |
+
```
|
| 31 |
+
|
| 32 |
+
#### (B) Single GPU
|
| 33 |
+
- ์ฑ๊ธ GPU์์๋ `accelerate launch ...` ๋์ `lerobot-train`์ ์ง์ ์คํํฉ๋๋ค.
|
| 34 |
+
|
| 35 |
+
```bash
|
| 36 |
+
CUDA_VISIBLE_DEVICES=0 \
|
| 37 |
+
lerobot-train \
|
| 38 |
+
--wandb.enable=true \
|
| 39 |
+
--wandb.project=lerobot \
|
| 40 |
+
--dataset.repo_id=${HF_USER}/${TASK_NAME} \
|
| 41 |
+
--output_dir=${HF_USER}/pi05_pickplace \
|
| 42 |
+
--policy.type=pi05 \
|
| 43 |
+
--policy.pretrained_path=lerobot/pi05_base \
|
| 44 |
+
--policy.dtype=bfloat16 \
|
| 45 |
+
--policy.device=cuda \
|
| 46 |
+
--steps=3000 \
|
| 47 |
+
--batch_size=64 \
|
| 48 |
+
--log_freq=100 \
|
| 49 |
+
--save_checkpoint=true \
|
| 50 |
+
--save_freq=500 \
|
| 51 |
+
--policy.push_to_hub=true \
|
| 52 |
+
--policy.repo_id=${HF_USER}/pi05_pickplace
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
| ์ต์
๋ช
| ์ค๋ช
| ๋น๊ณ |
|
| 56 |
+
|---|---|---|
|
| 57 |
+
| `--policy.device` | ์ฌ์ฉ device | |
|
| 58 |
+
| `--policy.dtype` | ํ์ต dtype | |
|
| 59 |
+
| `--steps` | ํ์ต ์คํ
์ | |
|
| 60 |
+
| `--batch_size` | ํ๋ก์ธ์ค๋น ๋ฐฐ์น ์ฌ์ด์ฆ | |
|
| 61 |
+
| `--optimizer.grad_clip_norm` | gradient clipping max norm | ์ด๋ฐ ๋ฐ์ฐ/NaN์ ๋ง๋ ์์ ์ฅ์น. ๋๋ฌด ์์ผ๋ฉด ํ์ต์ด ๋ต๋ตํด์ง ์ ์์ด โ์์ ํ โ ํ์ ์ ์ํโ ์์๋ก ์กฐ์ ํ๋ ๊ฒ ๋ฌด๋ํจ. |
|
| 62 |
+
| `--num_workers` | DataLoader ์์ปค ์ | |
|
| 63 |
+
| `--dataset.streaming` | ์คํธ๋ฆฌ๋ฐ ๋ฐ์ดํฐ์
๋ชจ๋ | ๋ฐ์ดํฐ๋ฅผ ๋ฏธ๋ฆฌ ๋ค cachingํ์ง ์๊ณ ์์ฐจ ๊ณต๊ธ๋ฐ๋ ๋ชจ๋. streaming์ ์
ํ/๋๋ค ์ ๊ทผ์ด ์ ํ๋๊ธฐ ์ฌ์ ํ์ต ๊ณก์ ์ด ๋ฌ๋ผ์ง ์ ์์. ๋ฐ์ดํฐ๊ฐ ๋๋ฌด ์ปค์ local cache๊ฐ ๋ถ๋ด์ธ ๊ฒฝ์ฐ ์ ๋ฆฌ. |
|
| 64 |
+
| `--policy.drop_n_last_frames` | ์ํผ์๋ ๋ง์ง๋ง Nํ๋ ์ ์ ๊ฑฐ | |
|
| 65 |
+
| `--wandb.enable` | W&B ๋ก๊น
์ฌ์ฉ ์ฌ๋ถ | |
|
| 66 |
+
| `--wandb.project` | W&B ํ๋ก์ ํธ๋ช
| |
|
| 67 |
+
| `--job_name` | ํ์ต ์์
์ด๋ฆ | |
|
| 68 |
+
| `--output_dir` | ๊ฒฐ๊ณผ/์ฒดํฌํฌ์ธํธ ์ถ๋ ฅ ๊ฒฝ๋ก | |
|
| 69 |
+
| `--save_checkpoint` | ์ฒดํฌํฌ์ธํธ ์ ์ฅ ์ฌ๋ถ | |
|
| 70 |
+
| `--save_freq` | ์ฒดํฌํฌ์ธํธ ์ ์ฅ ์ฃผ๊ธฐ(step) | |
|
| 71 |
+
| `--seed` | ๋๋ค ์๋ | |
|
| 72 |
+
| `--resume` | ์ฒดํฌํฌ์ธํธ์์ ์ฌ์์ | |
|
| 73 |
+
| `--checkpoint_path` | ์ฌ์์ ์ฒดํฌํฌ์ธํธ ๊ฒฝ๋ก | |
|
| 74 |
+
| `--rename_map` | observation ํค ์ด๋ฆ ๋งคํ | ๋ฐ์ดํฐ์
์ ์นด๋ฉ๋ผ ์ด๋ฆ๊ณผ policy๊ฐ ๊ธฐ๋ํ๋ ์นด๋ฉ๋ผ ์ด๋ฆ์ด ๋ค๋ฅผ ๋ ์ฌ์ฉ. |
|
| 75 |
+
| `--eval_freq` | ํ์ต ์ค ํ๊ฐ ์ฃผ๊ธฐ(step) | `eval_freq > 0`์ด์ด๋ env ์ค์ ์ด ํจ๊ป ์์ด์ผ ํ๊ฐ ํ๊ฒฝ์ด ๋ง๋ค์ด์ง. |
|
| 76 |
+
| `--eval.batch_size` | ํ๊ฐ์ฉ env ๊ฐ์(n_envs) | |
|
| 77 |
+
| `--eval.use_async_envs` | ๋น๋๊ธฐ ํ๊ฐ env ์ฌ์ฉ | |
|
| 78 |
+
| `--eval.n_episodes` | ํ๊ฐ ์ํผ์๋ ์ | |
|
| 79 |
+
| `--env.task` | (์๋ฎฌ) ํ๊ฐ ํ์คํฌ | ์๋ฎฌ๋ ์ด์
ํ๊ฐ์์ ์ด๋ค ํ์คํฌ๋ฅผ ๋๋ฆด์ง ์ง์ . |
|
| 80 |
+
| `--env.max_parallel_tasks` | (์๋ฎฌ) ํ๊ฐ ๋ณ๋ ฌ ์ ํ | |
|
| 81 |
+
|
| 82 |
+
|
| 83 |
+
### ํ์ต ์ฌ์์(resume) ์์
|
| 84 |
+
```bash
|
| 85 |
+
lerobot-train \
|
| 86 |
+
--resume=true \
|
| 87 |
+
--checkpoint_path="<checkpoint_dir>" ... (๋๋จธ์ง ๋์ผ)
|
| 88 |
+
```
|
| 89 |
+
|
| 90 |
+
### ํ์ต ์ค ์๋ฎฌ๋ ์ด์
ํ๊ฐ ์ผ๊ธฐ
|
| 91 |
+
ํ์ต ์ค ํ๊ฐ๋ฅผ ๋๋ฆฌ๋ ค๋ฉด `--eval_freq`๋ฅผ ์์๋ก ๋๊ณ , env ์ค์ ๊ณผ eval ์ค์ ์ ํจ๊ป ์ง์ ํฉ๋๋ค.
|
| 92 |
+
|
| 93 |
+
```bash
|
| 94 |
+
lerobot-train \
|
| 95 |
+
--eval_freq=500 \
|
| 96 |
+
--env.task="<your_env_task>" \
|
| 97 |
+
--env.max_parallel_tasks=8 \
|
| 98 |
+
--eval.batch_size=8 \
|
| 99 |
+
--eval.use_async_envs=true \
|
| 100 |
+
--eval.n_episodes=50 ... (๋๋จธ์ง ๋์ผ)
|
| 101 |
+
```
|
| 102 |
+
|
| 103 |
+
### (Optional) [RA-BC (Reward-Aligned Behavior Cloning)](https://huggingface.co/docs/lerobot/sarm)
|
| 104 |
+
|
| 105 |
+
RA-BC๋ โ๊ทธ๋ฅ ์์ฐ์ ๊ทธ๋๋ก ๋ฐ๋ผ ๋ฐฐ์ฐ๋(BC)โ ๋์ , **์งํ์ด ์ข์์ง๋(=์ฑ๊ณต์ ๊ฐ๊น์์ง๋) ๊ตฌ๊ฐ์ ๋ ๊ฐํ๊ฒ ํ์ต**ํ๋๋ก ์ํ๋ณ ๊ฐ์ค์น๋ฅผ ์ ์ฉํ๋ ๋ฐฉ์์
๋๋ค.
|
| 106 |
+
LeRobot ํ์ต ์ฝ๋์์๋ RA-BC๋ฅผ ์ผ๋ฉด ๋ฐฐ์น ์์์ ์ํ๋ง๋ค weight๋ฅผ ์ ์ฉํด **๊ฐ์ค ํ๊ท loss**๋ก ์
๋ฐ์ดํธํ๋ ํ๋ฆ์ ํ๋๋ค.
|
| 107 |
+
|
| 108 |
+
| ์ต์
๋ช
| ์ค๋ช
| ๋ถ๊ฐ ์ค๋ช
(์ธ์ /์ ์ฐ๋์ง) |
|
| 109 |
+
|---|---|---|
|
| 110 |
+
| `--use_rabc` | RA-BC ์ฌ์ฉ ์ฌ๋ถ | |
|
| 111 |
+
| `--rabc_progress_path` | progress/๊ฐ์ค์น ํ์ผ ๊ฒฝ๋ก | ์ฌ์ ์ ๊ณ์ฐ๋ progress/weight ํ์ผ ํ์. |
|
| 112 |
+
| `--rabc_head_mode` | progress head ์ ํ | progress ์์ธก์ ์ด๋ค head๋ฅผ ์ฌ์ฉํ ์ง ์ ํ. |
|
| 113 |
+
| `--rabc_kappa` | ๊ฐ์ค์น ๋ฏผ๊ฐ๋(์ธ๊ธฐ) | ๊ฐ์ด ์ปค์ง์๋ก ํน์ ๊ตฌ๊ฐ์ ๊ณผ๋ํ๊ฒ ์ง์คํ ์ ์์. |
|
| 114 |
+
| `--rabc_epsilon` | ์์น ์์ ํ ์์ | ๋ถ๋ชจ 0 ๋ฐฉ์ง |
|
| 115 |
+
|
| 116 |
+
---
|
| 117 |
+
|
| 118 |
+
## 4-2. Run inference
|
| 119 |
+
|
| 120 |
+
### ๋ฐฉ๋ฒ 1: Inference by `lerobot-record`
|
| 121 |
+
```bash
|
| 122 |
+
CUDA_VISIBLE_DEVICES=1 \
|
| 123 |
+
lerobot-record \
|
| 124 |
+
--robot.type=so101_follower \
|
| 125 |
+
--robot.port=/dev/ttyACM0 \
|
| 126 |
+
--robot.id=follower_new_gripper \
|
| 127 |
+
--robot.cameras="{
|
| 128 |
+
top: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30},
|
| 129 |
+
wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}
|
| 130 |
+
}" \
|
| 131 |
+
--dataset.repo_id=${HF_USER}/eval_pi05 \ # The name of dataset begins by eval to reflect that you are running inference (e.g. ${HF_USER}/eval_pi05)
|
| 132 |
+
--dataset.push_to_hub=false \
|
| 133 |
+
--dataset.single_task="pick a cup and place on the pink circle." \
|
| 134 |
+
--policy.path=${HF_USER}/pi05_pickplace \
|
| 135 |
+
--display_data=true
|
| 136 |
+
```
|
| 137 |
+
|
| 138 |
+
### ์ฃผ์ ์ต์
์ค๋ช
|
| 139 |
+
| ์ต์
๋ช
| ์ค๋ช
|
|
| 140 |
+
|---|---|
|
| 141 |
+
| `--robot.type` | follower ๋ก๋ด ํ์
|
|
| 142 |
+
| `--robot.port` | follower USB ํฌํธ |
|
| 143 |
+
| `--robot.id` | follower ๋ก๋ด ๊ณ ์ ID (calibration ๋ ์ค์ ํ ID) |
|
| 144 |
+
| `--robot.cameras` | ์นด๋ฉ๋ผ ์ ๋ณด |
|
| 145 |
+
| `--dataset.repo_id` | ๊ธฐ๋ก/ํ๊ฐ ๋ฐ์ดํฐ๋ฅผ ์ ์ฅํ HF ๋ฐ์ดํฐ์
repo |
|
| 146 |
+
| `--dataset.push_to_hub` | Huggingface ์๋ ์
๋ก๋ ์ฌ๋ถ |
|
| 147 |
+
| `--dataset.single_task` | language instruction |
|
| 148 |
+
| `--policy.path` | ์ฌ์ฉํ ์ ์ฑ
(๋ชจ๋ธ) ๊ฒฝ๋ก |
|
| 149 |
+
| `--display_data` | ํ๋ฉด์ ๋ฐ์ดํฐ ํ์ ์ฌ๋ถ (Rerun) |
|
| 150 |
+
|
| 151 |
+
---
|
| 152 |
+
|
| 153 |
+
### ๋ฐฉ๋ฒ 2: [Asynchronous inference (PolicyServer + RobotClient)](https://huggingface.co/docs/lerobot/async)
|
| 154 |
+
|
| 155 |
+
๋น๋๊ธฐ(Async) ์ถ๋ก ์ **action prediction(์๋ฒ)** ๊ณผ **action execution(๋ก๋ด)** ์ ๋ถ๋ฆฌํฉ๋๋ค.
|
| 156 |
+
๋ก๋ด์ ์ด๋ฏธ ๋ฐ์ action chunk๋ฅผ ์คํํ๋ ๋์, ์๋ฒ๋ ๋ค์ chunk๋ฅผ ๋ฏธ๋ฆฌ ๊ณ์ฐํด โ๋ฉ์นซ๊ฑฐ๋ฆผ(wait-for-inference)โ์ ์ค์
๋๋ค.
|
| 157 |
+
|
| 158 |
+
#### 0) ์ค์น
|
| 159 |
+
|
| 160 |
+
```bash
|
| 161 |
+
pip install -e ".[async]"
|
| 162 |
+
```
|
| 163 |
+
|
| 164 |
+
---
|
| 165 |
+
|
| 166 |
+
#### 1) Policy Server ์คํ (์๋ฒ/๋ณ๋ ํฐ๋ฏธ๋)
|
| 167 |
+
|
| 168 |
+
`PolicyServer`๋ ์ฒ์์๋ โ๋น ์ปจํ
์ด๋โ๋ก ๋จ๊ณ , **์ฒซ ์ฐ๊ฒฐ(handshake) ๋ ํด๋ผ์ด์ธํธ๊ฐ ์ด๋ค policy๋ฅผ ์ด๋ค device๋ก ๋๋ฆด์ง**๋ฅผ ์ ๋ฌํฉ๋๋ค.
|
| 169 |
+
|
| 170 |
+
```bash
|
| 171 |
+
# ๊ธฐ๋ณธ ์์ (localhost:8080)
|
| 172 |
+
CUDA_VISIBLE_DEVICES=0,1,2,3 \
|
| 173 |
+
python -m lerobot.async_inference.policy_server \
|
| 174 |
+
--host=127.0.0.1 \
|
| 175 |
+
--port=8080
|
| 176 |
+
```
|
| 177 |
+
|
| 178 |
+
---
|
| 179 |
+
|
| 180 |
+
| ์ต์
๋ช
| ์ค๋ช
| ๋ถ๊ฐ ์ค๋ช
|
|
| 181 |
+
|---|---|---|
|
| 182 |
+
| `--host` | ์๋ฒ ๋ฐ์ธ๋ฉ ์ฃผ์ | ๊ฐ์ ๋จธ์ ์ด๋ฉด `127.0.0.1`/`localhost`. ๋คํธ์ํฌ๋ก ๋ถ์ผ ๊ฑฐ๋ฉด `0.0.0.0` ๋๋ ์๋ฒ IP๋ฅผ ์ฌ์ฉ. |
|
| 183 |
+
| `--port` | ์๋ฒ ํฌํธ | ๊ธฐ๋ณธ `8080`. |
|
| 184 |
+
| `--fps` | ์๋ฒ ๊ธฐ์ค FPS | ๊ธฐ๋ณธ 30. `environment_dt = 1/fps`๋ก ๋ด๋ถ timestep์ ๊ณ์ฐ. |
|
| 185 |
+
| `--inference_latency` | ๋ชฉํ ์ถ๋ก ์ง์ฐ(์ด) | ๊ธฐ๋ณธ `1/fps`. ์๋ฒ๋ action chunk๋ฅผ ๋ง๋ ๋ค, ์ด ๊ฐ์ ๋ง์ถฐ pacing(sleep)ํฉ๋๋ค. |
|
| 186 |
+
| `--obs_queue_timeout` | observation queue timeout(์ด) | ๊ธฐ๋ณธ 2์ด. timeout ๋ด์ obs๊ฐ ์์ผ๋ฉด ์๋ฒ๋ ๋น ์๋ต์ ๋ฐํ. |
|
| 187 |
+
|
| 188 |
+
> ์ฐธ๊ณ : ์๋ฒ๋ observation queue๋ฅผ `maxsize=1`๋ก ๋๊ณ โ๊ฐ์ฅ ์ต์ obsโ ์์ฃผ๋ก ์ฒ๋ฆฌํฉ๋๋ค.
|
| 189 |
+
|
| 190 |
+
---
|
| 191 |
+
|
| 192 |
+
#### 2) Robot Client ์คํ (๋ก๋ด/๋ณ๋ ํฐ๋ฏธ๋)
|
| 193 |
+
|
| 194 |
+
ํด๋ผ์ด์ธํธ๋ ๋ก๋ด์์ observation์ ์ฝ์ด ์๋ฒ๋ก ๋ณด๋ด๊ณ , ์๋ฒ๊ฐ ๋ง๋ action chunk๋ฅผ ๋ฐ์ **๋ก์ปฌ ํ์ ์์ ์คํ**ํฉ๋๋ค.
|
| 195 |
+
|
| 196 |
+
```bash
|
| 197 |
+
python -m lerobot.async_inference.robot_client \
|
| 198 |
+
--server_address=127.0.0.1:8080 \
|
| 199 |
+
--robot.type=so101_follower \
|
| 200 |
+
--robot.port=/dev/so101_follower \
|
| 201 |
+
--robot.id=follower \
|
| 202 |
+
--robot.cameras='{
|
| 203 |
+
top: {type: opencv, index_or_path: "/dev/video0", width: 640, height: 480, fps: 30},
|
| 204 |
+
wrist: {type: opencv, index_or_path: "/dev/video2", width: 640, height: 480, fps: 30}
|
| 205 |
+
}' \
|
| 206 |
+
--task="pick a cup and place on the pink circle." \
|
| 207 |
+
--policy_type=pi05 \
|
| 208 |
+
--pretrained_name_or_path=${HF_USER}/pi05_pickplace \
|
| 209 |
+
--policy_device=cuda \
|
| 210 |
+
--actions_per_chunk=50 \
|
| 211 |
+
--chunk_size_threshold=0.7 \
|
| 212 |
+
--aggregate_fn_name=weighted_average \
|
| 213 |
+
--debug_visualize_queue_size=True
|
| 214 |
+
```
|
| 215 |
+
|
| 216 |
+
| ์ต์
๋ช
| ์ค๋ช
| ๋ถ๊ฐ ์ค๋ช
|
|
| 217 |
+
|---|---|---|
|
| 218 |
+
| `--server_address` | ์๋ฒ ์ฃผ์ | `host:port` ํํ. |
|
| 219 |
+
| `--robot.type` | ๋ก๋ด ํ์
| |
|
| 220 |
+
| `--robot.port` | ๋ก๋ด ํฌํธ | |
|
| 221 |
+
| `--robot.id` | ๋ก๋ด ID | |
|
| 222 |
+
| `--robot.cameras` | ์นด๋ฉ๋ผ ์ค์ | **ํค ์ด๋ฆ์ด policy๊ฐ ๊ธฐ๋ํ๋ ์ด๋ฏธ์ง ํค**์ ๋ง์์ผ ํจ. (ํค ๋ถ์ผ์น ์ ์ด๋ฏธ์ง๊ฐ โ์๋ ๊ฒ์ฒ๋ผโ ๋ค์ด๊ฐ๊ฑฐ๋ ์ ์ฒ๋ฆฌ์์ ๋ฌธ์ ๋ฐ์) |
|
| 223 |
+
| `--task` | language instruction | |
|
| 224 |
+
| `--policy_type` | ์ ์ฑ
ํ์
| |
|
| 225 |
+
| `--pretrained_name_or_path` | ๋ชจ๋ธ ๊ฒฝ๋ก | |
|
| 226 |
+
| `--policy_device` | (์๋ฒ) ์ ์ฑ
์คํ device | ์ด ๊ฐ์ โํด๋ผ์ด์ธํธ ์ต์
โ์ด์ง๋ง, ์ค์ ๋ก๋ **์๋ฒ ์ชฝ์์** ๋ชจ๋ธ์ ์ฌ๋ฆด device๋ฅผ ์ง์ ํจ. |
|
| 227 |
+
| `--actions_per_chunk` | ํ ๋ฒ์ ๋ฐ์ action ์ | ๊ถ์ฅ ๋ฒ์๋ ๋ณดํต 10~50. ๊ฐ์ด ํฌ๋ฉด ๋๊น์ด ์ค ์ ์์ง๋ง, ๋ ๋จผ horizon ์์ธก์ผ๋ก ์ค์ฐจ ๋์ ๊ฐ๋ฅ์ฑ์ด ์ปค์ง. |
|
| 228 |
+
| `--chunk_size_threshold` | obs ์ ์ก ํธ๋ฆฌ๊ฑฐ | ํ๊ฐ โ์ผ๋ง๋ ๋น์์ ๋โ ์ obs๋ฅผ ๋ณด๋ด ๋ค์ chunk๋ฅผ ๋ฏธ๋ฆฌ ๊ณ์ฐํ ์ง ๊ฒฐ์ . ๊ฐ์ด ํด์๋ก ์ ์์ฑโ(๋ ์์ฃผ ์
๋ฐ์ดํธ) / ์๋ฒ ๋ถ๋ดโ. |
|
| 229 |
+
| `--aggregate_fn_name` | action ๊ฒน์นจ ๊ตฌ๊ฐ ๋ณํฉ ํจ์ | action chunk๊ฐ ๊ฒน์น๋ฉด ๊ฐ์ timestep์ action์ ์์. (์: `weighted_average`, `latest_only`, `average`, `conservative`) |
|
| 230 |
+
| `--debug_visualize_queue_size` | queue ํฌ๊ธฐ ์๊ฐํ | ์ข
๋ฃ ์ queue ํฌ๊ธฐ ๋ณํ๋ฅผ plotting |
|
| 231 |
+
| `--fps` | ํด๋ผ์ด์ธํธ ์ ์ด ๋ฃจํ FPS | ๊ธฐ๋ณธ 30. `environment_dt=1/fps`๋ก sleep์ ์กฐ์ . |
|
| 232 |
+
|
| 233 |
+
---
|
| 234 |
+
|
| 235 |
+
#### 3) ํ๋ ํ (actions_per_chunk / chunk_size_threshold)
|
| 236 |
+
|
| 237 |
+
- `actions_per_chunk` โ: actions ๊ณ ๊ฐ ์ํ โ โ ๋๊น โ / ํ์ง๋ง ๊ธด horizon ์์ธก โ ์ค์ฐจ ๋์ ๊ฐ๋ฅ์ฑ โ
|
| 238 |
+
- `chunk_size_threshold` โ: ๋ ์์ฃผ ์ chunk ์์ฒญ โ ์ ์์ฑ โ / ํ์ง๋ง ์๋ฒ ์์ฒญ๋ โ โ ์ถ๋ก ํ์ดํ๋ผ์ธ ๋ถ๋ด โ
|
| 239 |
+
|
| 240 |
+
---
|
| 241 |
+
|
| 242 |
+
#### 4) ์์ฃผ ๋๋ ๋ฌธ์
|
| 243 |
+
- **์๋ฒ๋ ๋จ๋๋ฐ ๋ก๋ด์ด ์ ์์ง์**: `server_address`๊ฐ ์๋ฒ์ `host:port`์ ์ผ์นํ๋์ง, ๋ฐฉํ๋ฒฝ/ํฌํธ๊ฐ ๋งํ์์ง ์์์ง ํ์ธ
|
| 244 |
+
- **์นด๋ฉ๋ผ ํค ๋งค์นญ ์ค๋ฅ**: `--robot.cameras`์ ํค ์ด๋ฆ์ด policy๊ฐ ๊ธฐ๋ํ๋ ์ด๋ฏธ์ง ํค์ ์ผ์นํด์ผ ํจ
|
| 245 |
+
- **Stop-and-go**: `actions_per_chunk`๋ฅผ ๋๋ฆฌ๊ฑฐ๋ `chunk_size_threshold`๋ฅผ ํค์ฐ๋, ์๋ฒ ๋ฆฌ์์ค/๋คํธ์ํฌ ์ง์ฐ์ ํจ๊ป ๊ณ ๋ ค
|
| 246 |
+
|