Robotics
LeRobot
Safetensors
act
omx
depth
moge

OMX ํ•‘ํ‚ค ์œ„ ๋ฐ•์Šค๋ฅผ ์„ ๋ฐ˜์— ๋„ฃ๊ธฐ ยท ACT ๊ธฐ๋ณธ / ์ „๋ฅ˜ + MoGe (2026-11-20)

ํ•‘ํ‚ค ์œ„์— ๋†“์ธ ๋ฐ•์Šค๋ฅผ ์ง‘์–ด ์„ ๋ฐ˜ 2์—ด ๊ฐ™์€ ์ž๋ฆฌ์— ๋„ฃ๋Š” ์‹œ์—ฐ์œผ๋กœ ACT๋ฅผ ํ•™์Šตํ•œ ๋ชจ๋ธ๊ณผ ์‹คํ–‰ ์ฝ”๋“œ์ž…๋‹ˆ๋‹ค.

  • ๋ฐ์ดํ„ฐ: Sungah/omx_1119_sakim_success โ€” ์‹œ์—ฐ 10๊ฐœ, 11,936ํ”„๋ ˆ์ž„, 30 fps, ์นด๋ฉ”๋ผ top(์œ„์ชฝ)ยทwrist(์†๋ชฉ)
  • 0๋ฒˆ ์‹œ์—ฐ์€ val loss ๊ณ„์‚ฐ์šฉ์œผ๋กœ ํ•™์Šต์—์„œ ๋บ์Šต๋‹ˆ๋‹ค. ํ•™์Šต์—๋Š” ์‹œ์—ฐ 9๊ฐœ(10,741ํ”„๋ ˆ์ž„)๋ฅผ ์ผ์Šต๋‹ˆ๋‹ค.
  • ์ž‘์—… ๋ฌธ์žฅ์€ ์“ฐ์ง€ ์•Š์Šต๋‹ˆ๋‹ค(ACT๋Š” ์ž‘์—… ๋ฌธ์žฅ์„ ์ž…๋ ฅ์œผ๋กœ ๋ฐ›์ง€ ์•Š์Œ).

๋“ค์–ด ์žˆ๋Š” ๋ชจ๋ธ

๋ฒˆํ˜ธ ํด๋” ์ž…๋ ฅ step
1โ€“8 act_1119_P0_base_<N>step top + wrist, state 6 (๊ด€์ ˆ 5 + ๊ทธ๋ฆฌํผ ์œ„์น˜) 500~4000
9โ€“16 act_1119_P1_grip_moge_<N>step top + top_depth + wrist, state 7 (+ ๊ทธ๋ฆฌํผ ์ „๋ฅ˜) 500~4000
  • ๋ฒˆํ˜ธ = ./infer.sh <๋ฒˆํ˜ธ>์— ์“ฐ๋Š” ๊ฐ’์ž…๋‹ˆ๋‹ค. ๊ฐ ์ค„ ์•ˆ์—์„œ 500 step ์ˆœ์„œ์ž…๋‹ˆ๋‹ค(์˜ˆ: 1 = P0 500 step, 8 = P0 4000 step, 16 = P1 4000 step).
  • top_depth: ์œ„์ชฝ ์นด๋ฉ”๋ผ ์˜์ƒ์„ MoGe-2 ViT-B(ํ† ํฐ 1200)๋กœ ์ถ”์ •ํ•œ ๊ฑฐ๋ฆฌ๋ฅผ 0.87~2.68 m ๋ฒ”์œ„์˜ ํ‘๋ฐฑ ์˜์ƒ์œผ๋กœ ๋งŒ๋“  ๊ฒƒ์ž…๋‹ˆ๋‹ค(๊ฐ€๊นŒ์šธ์ˆ˜๋ก ํฐ์ƒ‰). ์‹คํ–‰ํ•  ๋•Œ ์‹ค์‹œ๊ฐ„์œผ๋กœ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
  • ๊ทธ๋ฆฌํผ ์ „๋ฅ˜: ๋ฐ•์Šค๋ฅผ ์ฅ๋ฉด ํฌ๊ฒŒ ๋‚ด๋ ค๊ฐ€๋Š” ๊ฐ’์ด๋ผ "์žก์•˜๋Š”์ง€"๋ฅผ ์•Œ๋ ค ์ค๋‹ˆ๋‹ค.
  • ์‹ค๋ฌผ ์„ฑ๊ณต๋ฅ : (ํ…Œ์ŠคํŠธ ํ›„ ๊ธฐ๋ก)

val loss

val loss = ํ•™์Šต์— ์“ฐ์ง€ ์•Š์€ 0๋ฒˆ ์‹œ์—ฐ์—์„œ, ๋ชจ๋ธ์ด ์˜ˆ์ธกํ•œ ๋™์ž‘ 100์Šคํ…์ด ์‹œ์—ฐ๊ณผ ์–ผ๋งˆ๋‚˜ ๋‹ค๋ฅธ์ง€(L1, ์ •๊ทœํ™” ๋‹จ์œ„, ๋‚ฎ์„์ˆ˜๋ก ์‹œ์—ฐ๊ณผ ๋น„์Šท). ์‹ค๋ฌผ์—์„œ ๋Œ๋ฆด ๋•Œ์ฒ˜๋Ÿผ ๊ณ„์‚ฐํ–ˆ์Šต๋‹ˆ๋‹ค(๋ฌด์ž‘์œ„ ์š”์†Œยท์ด๋ฏธ์ง€ ์ฆ๊ฐ• ์—†์Œ). ํ•™์Šต ๋กœ๊ทธ์˜ loss์™€๋Š” ๊ณ„์‚ฐ์ด ๋‹ฌ๋ผ ๋ฐ”๋กœ ๋น„๊ตํ•  ์ˆ˜ ์—†์Šต๋‹ˆ๋‹ค.

  • ํ•™์Šต ์‹œ์—ฐ = ํ•™์Šต์— ์“ด ์‹œ์—ฐ 1ยท5ยท9๋ฒˆ์˜ ํ‰๊ท  โ€” ๋น„๊ต์šฉ
  • ๊ตต์€ ๊ฐ’ = ๊ทธ ๋ชจ๋ธ์—์„œ val์ด ๊ฐ€์žฅ ๋‚ฎ์€ step
๋ฒˆํ˜ธ ๋ชจ๋ธ step ํ•™์Šต ์‹œ์—ฐ(1ยท5ยท9 ํ‰๊ท ) val(0๋ฒˆ)
1 P0 ๊ธฐ๋ณธ 500 0.128 0.312
2 P0 ๊ธฐ๋ณธ 1000 0.101 0.318
3 P0 ๊ธฐ๋ณธ 1500 0.097 0.324
4 P0 ๊ธฐ๋ณธ 2000 0.094 0.327
5 P0 ๊ธฐ๋ณธ 2500 0.087 0.333
6 P0 ๊ธฐ๋ณธ 3000 0.085 0.331
7 P0 ๊ธฐ๋ณธ 3500 0.084 0.328
8 P0 ๊ธฐ๋ณธ 4000 0.084 0.331
9 P1 ์ „๋ฅ˜ + MoGe 500 0.119 0.305
10 P1 ์ „๋ฅ˜ + MoGe 1000 0.114 0.312
11 P1 ์ „๋ฅ˜ + MoGe 1500 0.104 0.317
12 P1 ์ „๋ฅ˜ + MoGe 2000 0.096 0.318
13 P1 ์ „๋ฅ˜ + MoGe 2500 0.090 0.310
14 P1 ์ „๋ฅ˜ + MoGe 3000 0.087 0.318
15 P1 ์ „๋ฅ˜ + MoGe 3500 0.085 0.317
16 P1 ์ „๋ฅ˜ + MoGe 4000 0.085 0.317
  • ํ•™์Šต ์‹œ์—ฐ loss๋Š” step์ด ๋Š˜์ˆ˜๋ก ๊ณ„์† ๋‚ด๋ ค๊ฐ€์ง€๋งŒ, val(0๋ฒˆ)์€ 0.30~0.33์—์„œ ๊ฑฐ์˜ ๊ทธ๋Œ€๋กœ์ž…๋‹ˆ๋‹ค. ์‹œ์—ฐ์ด 9๊ฐœ๋ฟ์ด๋ผ ์ฒ˜์Œ ๋ณด๋Š” ์‹œ์—ฐ์œผ๋กœ๋Š” ์ž˜ ์ผ๋ฐ˜ํ™”๋˜์ง€ ์•Š๋Š” ์ƒํƒœ์ž…๋‹ˆ๋‹ค.
  • P1์ด P0๋ณด๋‹ค val์ด ๋ชจ๋“  step์—์„œ ์กฐ๊ธˆ(0.006~0.023) ๋‚ฎ์Šต๋‹ˆ๋‹ค.
  • 0๋ฒˆ ์‹œ์—ฐ์€ ๋‹ค๋ฅธ ์‹œ์—ฐ๋ณด๋‹ค ๊ทธ๋ฆฌํผ๋ฅผ ์•ฝํ•˜๊ฒŒ ์ฅ์—ˆ์Šต๋‹ˆ๋‹ค(์ „๋ฅ˜ ์ตœ์†Œ โˆ’129, ๋‹ค๋ฅธ ์‹œ์—ฐ์€ โˆ’150 ์•„๋ž˜). val loss๊ฐ€ ๋†’์€ ๋ฐ๋Š” ์ด ์ฐจ์ด๋„ ์˜ํ–ฅ์„ ์ค„ ์ˆ˜ ์žˆ์œผ๋‹ˆ, ์‹ค๋ฌผ ํ…Œ์ŠคํŠธ๋กœ ํ™•์ธํ•ด์•ผ ํ•ฉ๋‹ˆ๋‹ค.

ํ•™์Šต ์ •๋ณด (lerobot 0.4.4)

ํ•ญ๋ชฉ ๊ฐ’
์‹œ์ž‘์  ResNet18 ImageNet ๊ฐ€์ค‘์น˜
batch ยท step 256 (32ร—8๋กœ ๋‚˜๋ˆ  ๊ณ„์‚ฐ) ยท 4000 (ํ•™์Šต ๋ฐ์ดํ„ฐ ์•ฝ 95ํšŒ ๋ฐ˜๋ณต), 500๋งˆ๋‹ค ์ €์žฅ
ํ•™์Šต๋ฅ  cosine ์ตœ๋Œ€ 4e-5 (์ด๋ฏธ์ง€ ์ธ์ฝ”๋” 1e-5), warmup 100 โ†’ 1e-6
chunk 100
๊ธฐํƒ€ ์ด๋ฏธ์ง€ ์ฆ๊ฐ•(์ƒ‰ยท์„ ๋ช…๋„ยทํšŒ์ „ยท์ด๋™), grad clip 10
๋งˆ์ง€๋ง‰ ํ•™์Šต loss P0 0.066, P1 0.063

๋‹ค์‹œ ๋งŒ๋“ค๊ธฐ: ์„ค์น˜ โ†’ ๋ฐ์ดํ„ฐ ๋ฐ›๊ธฐยท๋งŒ๋“ค๊ธฐ โ†’ ํ•™์Šต โ†’ val loss โ†’ ์ถ”๋ก 

training/ ํด๋”์— ํ•™์Šต ๋ฐ์ดํ„ฐ๋ฅผ ๋งŒ๋“  ์ฝ”๋“œ, ํ•™์Šต ์Šคํฌ๋ฆฝํŠธ, val loss ๊ณ„์‚ฐ ์ฝ”๋“œ๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค. ์ž์„ธํ•œ ์„ค๋ช…์€ training/README.md๋ฅผ ๋ณด์„ธ์š”.

# 0. ์„ค์น˜
pip install "lerobot[dynamixel]==0.4.4"
pip install opencv-python-headless scipy trimesh       # MoGe (P1)
pip install --no-deps git+https://github.com/EasternJournalist/utils3d.git git+https://github.com/microsoft/MoGe.git

# 1. ์ด ์ €์žฅ์†Œ ๋ฐ›๊ธฐ
hf download Sungah/omx_1119_act --local-dir omx_1119_act
cd omx_1119_act

# 2. ๋ฐ์ดํ„ฐ ๋ฐ›๊ธฐ + ๋งŒ๋“ค๊ธฐ (์›๋ณธ์„ data/์— ๋ฐ›๊ณ  ํ•™์Šตยทval ๋ฐ์ดํ„ฐ 5๊ฐœ๋ฅผ ๋งŒ๋“ฆ, MoGe ๋‹จ๊ณ„๋Š” GPU ํ•„์š”)
DATA=$PWD/data bash training/make_datasets.sh

# 3. ํ•™์Šต
bash training/train_act.sh P0
bash training/train_act.sh P1

# 4. val loss
python -I training/tools/val_loss.py --dataset_root data/omx_1119_sakim_success --episodes 0,1,5,9 \
    --ckpts outputs/train/act_1119_P0_base/checkpoints --out outputs/eval/val_1119_P0.json
python -I training/tools/val_loss.py --dataset_root data/omx_1119_grip_moge --episodes 0,1,5,9 \
    --ckpts outputs/train/act_1119_P1_grip_moge/checkpoints --out outputs/eval/val_1119_P1.json

# 5. ์ถ”๋ก  (์ด ์ €์žฅ์†Œ์˜ ๋ชจ๋ธ๋กœ ๋ฐ”๋กœ)
bash infer.sh 16

์‹คํ–‰ ๋ฐฉ๋ฒ•

์ด ์ €์žฅ์†Œ์— ์‹คํ–‰ ์ฝ”๋“œ(lerobot 0.4.4์šฉ)๊ฐ€ ๊ฐ™์ด ๋“ค์–ด ์žˆ์Šต๋‹ˆ๋‹ค. ๋ฒˆํ˜ธ๋กœ ๋ชจ๋ธ์„ ๊ณจ๋ผ ์‹คํ–‰ํ•˜๊ณ , GPU(cuda) ๋กœ ์‹คํ–‰ํ•ฉ๋‹ˆ๋‹ค.

hf download Sungah/omx_1119_act --local-dir omx_1119_act
cd omx_1119_act
chmod +x infer.sh inference.sh   # hf download๋Š” ์‹คํ–‰ ๊ถŒํ•œ์„ ์ง€์›€ (์•ˆ ํ•˜๋ฉด bash infer.sh ๋กœ ์‹คํ–‰)
./infer.sh               # ๋ชจ๋ธ ๋ฒˆํ˜ธ ๋ชฉ๋ก
./infer.sh 8             # P0 ๊ธฐ๋ณธ 4000 step
./infer.sh 16            # P1 ๊ทธ๋ฆฌํผ ์ „๋ฅ˜ + MoGe 4000 step
  • MoGe ๋ชจ๋ธ(9~16)์€ ์ฒ˜์Œ ์‹คํ–‰ํ•  ๋•Œ Hugging Face์—์„œ Ruicheng/moge-2-vitb-normal ๊ฐ€์ค‘์น˜๋ฅผ ๋ฐ›์Šต๋‹ˆ๋‹ค. ์ธํ„ฐ๋„ท์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
  • ์‹œ์ž‘ ์ž์„ธ: ์‹œ์—ฐ์€ ๋ชจ๋‘ ๊ฐ™์€ ์ž์„ธ(๊ด€์ ˆ ์•ฝ [39, โˆ’65, 31, 34, โˆ’1], ๊ทธ๋ฆฌํผ 60)์—์„œ ์‹œ์ž‘ํ•ฉ๋‹ˆ๋‹ค. ๋น„์Šทํ•œ ์ž์„ธ์—์„œ ์‹œ์ž‘ํ•˜์„ธ์š”.

infer.sh๊ฐ€ ์ž๋™์œผ๋กœ ํ•˜๋Š” ์ผ

  • ๋กœ๋ด‡ ํƒ€์ž…์„ ๋ชจ๋ธ์— ๋งž์ถฅ๋‹ˆ๋‹ค.
    • P1(depth_info.json์ด ์žˆ์Œ): omx_follower_grip_depth โ€” ๊ทธ๋ฆฌํผ ์ „๋ฅ˜๋ฅผ state์— ๋ถ™์ด๊ณ , MoGe depth ์นด๋ฉ”๋ผ๋ฅผ ๋งŒ๋“ญ๋‹ˆ๋‹ค.
    • P0: omx_follower
  • MoGe ๋ชจ๋ธ์€ ์ •์ฑ…์ด ์ƒˆ๋กœ ์˜ˆ์ธกํ•˜๋Š” ํ”„๋ ˆ์ž„์—์„œ๋งŒ ์œ„์ชฝ ์นด๋ฉ”๋ผ ์˜์ƒ์œผ๋กœ depth๋ฅผ ๋ฐ”๋กœ ๋งŒ๋“ค์–ด ๋„ฃ์Šต๋‹ˆ๋‹ค.
    • ACT๋Š” n_action_steps=50์ด๋ผ 50ํ”„๋ ˆ์ž„(์•ฝ 1.7์ดˆ)๋งˆ๋‹ค ํ•œ ๋ฒˆ ์˜ˆ์ธกํ•ฉ๋‹ˆ๋‹ค. ๊ทธ ํ”„๋ ˆ์ž„๋งŒ depth ๊ณ„์‚ฐ ์‹œ๊ฐ„(RTX A6000์—์„œ ํ•œ ์žฅ์— ์•ฝ 31~46 ms)๋งŒํผ ๋Šฆ์–ด์ง‘๋‹ˆ๋‹ค.
    • ์˜ˆ์ „ ๋ฐฉ์‹(๋ฐฑ๊ทธ๋ผ์šด๋“œ์—์„œ ๊ณ„์† ๊ณ„์‚ฐ, 2~3ํ”„๋ ˆ์ž„ ๋Šฆ์Œ)์€ --robot.depth_sync=false๋กœ ์“ธ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ์‹คํ–‰ ์‹œ๊ฐ„์€ 60์ดˆ๊ฐ€ ๊ธฐ๋ณธ์ž…๋‹ˆ๋‹ค(์‹œ์—ฐ ๊ธธ์ด 36~43์ดˆ).

๊ณตํ†ต ์„ค์ •

  • ํฌํŠธยท์นด๋ฉ”๋ผ๋Š” inference.sh ๋งจ ์œ„ "ํ˜„์žฅ ์„ค์ •"์„ ๊ณ ์น˜๊ฑฐ๋‚˜ ํ™˜๊ฒฝ๋ณ€์ˆ˜๋กœ ๋„˜๊น๋‹ˆ๋‹ค.
    • ์˜ˆ: ROBOT_PORT=/dev/ttyACM1 TOP_CAM=/dev/video4 WRIST_CAM=/dev/video6 ./infer.sh 16
  • GPU๋ฅผ ๋ฐ”๊พธ๋ ค๋ฉด: POLICY_DEVICE=cuda:0 DEPTH_DEVICE=cuda:0 ./infer.sh 16
  • ํ•™์Šต ๋•Œ์™€ ๊ฐ™์•„์•ผ ํ•˜๋Š” ๊ฒƒ: fps(30), ์œ„์ชฝยท์†๋ชฉ ์นด๋ฉ”๋ผ ์œ„์น˜ยท๊ฐ๋„, ํ•‘ํ‚คยท์„ ๋ฐ˜ ๋ฐฐ์น˜
    • depth ๋ฒ”์œ„(0.87~2.68 m)๋„ ์นด๋ฉ”๋ผ๊ฐ€ ๊ฐ™์€ ์ž๋ฆฌ์— ์žˆ๋‹ค๋Š” ์ „์ œ์ž…๋‹ˆ๋‹ค.
  • ํ•œ ์Šคํ… ์ด๋™๋Ÿ‰์€ --robot.max_relative_target=10์œผ๋กœ ์ œํ•œ๋ผ ์žˆ์Šต๋‹ˆ๋‹ค.
  • ์‹คํ–‰๋งˆ๋‹ค ์งง์€ ํ‰๊ฐ€ ๊ธฐ๋ก์ด outputs/eval/ ์•„๋ž˜์— ์ €์žฅ๋ฉ๋‹ˆ๋‹ค(Hub ์—…๋กœ๋“œ ์•ˆ ํ•จ).

ํŒŒ์ผ

ํŒŒ์ผ ๋‚ด์šฉ
infer.sh ๋ฒˆํ˜ธ๋กœ ๋ชจ๋ธ ๊ณจ๋ผ ์‹คํ–‰
inference.sh lerobot-record ์˜ต์…˜ ์ „์ฒด(ํฌํŠธยท์นด๋ฉ”๋ผ ์„ค์ •)
omx_run.py lerobot-record + ์•„๋ž˜ ๋กœ๋ด‡ ํƒ€์ž… ๋“ฑ๋ก
omx_depth_robot.py omx_follower_grip_depth: ๊ทธ๋ฆฌํผ ์ „๋ฅ˜ + MoGe depth ์นด๋ฉ”๋ผ, ์˜ˆ์ธก ํ”„๋ ˆ์ž„์—์„œ๋งŒ ๊ณ„์‚ฐ
omx_grip_robot.py omx_follower_grip: state์— ๊ทธ๋ฆฌํผ ์ „๋ฅ˜ ์ถ”๊ฐ€
omx_box.py ๋ฐ•์Šค ๋ฒˆํ˜ธ one-hot (์ด ์ €์žฅ์†Œ ๋ชจ๋ธ์€ ์•ˆ ์”€, omx_run.py๊ฐ€ BOX๊ฐ€ ์žˆ์„ ๋•Œ๋งŒ import)
act_ensemble.py (์„ ํƒ) ACT k์Šคํ… temporal ensembling
training/ ํ•™์Šต ๋ฐ์ดํ„ฐ ๋งŒ๋“ค๊ธฐยทํ•™์Šตยทval loss ์Šคํฌ๋ฆฝํŠธ (training/README.md)
weights/resnet18-f37072fd.pth ResNet18 ImageNet ๊ฐ€์ค‘์น˜(์ธํ„ฐ๋„ท ์—†์ด ์‹คํ–‰ํ•  ๋•Œ)
models/*/omx_info.json ๊ทธ๋ฆฌํผ ์ „๋ฅ˜ ์‚ฌ์šฉ ์—ฌ๋ถ€(P0 false, P1 true)
models/act_1119_P1_*/depth_info.json depth ๋ฒ”์œ„ยทMoGe ๋ชจ๋ธ ์„ค์ •
Downloads last month

-

Downloads are not tracked for this model. How to track
Video Preview
loading

Dataset used to train Sungah/omx_1119_act