README.md

Jason's multi_task_dit — SC1 Final Checkpoint

成功率: 9/9 (100%) on SO-101 real robot (2026-04-17) Policy: multi_task_dit (lerobot 0.5.1 built-in, ~277M params)

環境

conda activate a   # Jason's env on idlab2
# lerobot 來自 /home/kunhsiang/jdk/lib/lerobot/
# CLIP weights: /home/kunhsiang/jdk/hf_weights/clip-vit-base-patch16/

訓練指令（完整復現）

cd /home/kunhsiang/jdk
CUDA_VISIBLE_DEVICES=1 lerobot-train \
  --dataset.root=datasets/sc1_newcolor \
  --dataset.repo_id=YOUR_DATASET \
  --output_dir=outputs/train/multi_task_dit_repro \
  --policy.type=multi_task_dit \
  --policy.device=cuda \
  --policy.horizon=32 \
  --policy.n_action_steps=24 \
  --policy.objective=diffusion \
  --policy.noise_scheduler_type=DDIM \
  --policy.num_train_timesteps=100 \
  --policy.num_inference_steps=20 \
  --policy.clip_sample=true \
  --policy.clip_sample_range=1.0 \
  --policy.hidden_dim=512 \
  --policy.num_layers=6 \
  --policy.num_heads=8 \
  --policy.repo_id=jedeka30/grasp_box \
  --wandb.enable=true \
  --wandb.project=act \
  --steps=50000 \
  --save_freq=5000 \
  --batch_size=24 \
  --num_workers=8 \
  --eval_freq=2000

超參數摘要

參數	值
Policy	multi_task_dit
Objective	diffusion (DDIM)
horizon	32
n_action_steps	24
num_inference_steps	20 (關鍵，比 10 好很多)
num_train_timesteps	100
hidden_dim	512
num_layers	6
num_heads	8
batch_size	24
steps	50,000
Dataset	sc1_newcolor (3色: red/blue/green, ~80 episodes)

Eval 指令

cd /home/kunhsiang/jdk

# SC1: red/green/blue 三色（自動循環）
bash 3kh.sh ckpts/final_ckpts/pretrained_model

# SC2/3: on/outside the plate
bash 2test.sh ckpts/final_ckpts/pretrained_model

硬體設定

設備	值
Robot	SO-101 follower
Port	/dev/ttyACM1
GPU	CUDA_VISIBLE_DEVICES=1
cam_front	/dev/video2 (640×480, 30fps, MJPG)
cam_gripper	/dev/video0 (640×480, 30fps, MJPG)
cam_top	/dev/video4 (640×480, 30fps, MJPG)
episode_time_s	90

評測結果

Task	SR	Date
grasp sc1 (9 trials: red×3 / green×3 / blue×3)	9/9 (100%)	2026-04-17
grasp sc4 (put on plate)	2/6 (33%)	2026-05-04

注意

• lerobot 的 multi_task_dit 有 offline CLIP hack: 讀 /home/kunhsiang/jdk/hf_weights/clip-vit-base-patch16/ 標準 lerobot 0.5.1 會從 HF 下載 openai/clip-vit-base-patch16（需網路）
• TRANSFORMERS_OFFLINE=1 + HF_DATASETS_OFFLINE=1 在訓練時設定（離線模式）
• 訓練完後 make reset 會把 robot 歸位（Makefile 在 jdk 根目錄）

相關 Repo

• Dataset: kunhsiang/jason-sc1-newcolor-dataset
• Eval scripts: 此 repo 內 scripts/ 目錄