README.md

metadata

license: apache-2.0
tags:
  - robotics
  - lerobot
  - act
  - so101
  - so-arm101
  - manipulation
  - pick-and-place
pipeline_tag: robotics
library_name: lerobot

SO-ARM101 pick-and-place — ACT, single task (cube)

An ACT (Action Chunking Transformer) policy trained from scratch on one SO-ARM101 tabletop task: pick up a cube and place it in the bin.

Read this first. This checkpoint does not clear our offline acceptance bar — on every one of its six joints, its best step still predicts motion with larger error amplitude than a "hold current pose" baseline on at least one axis (worst_ratio 1.681, needs < 1). It is published as a reference and comparison artifact, not a deployable policy. See Offline evaluation.

Task

Single task, no language conditioning. LeRobot's ACT implementation takes no language input — which is exactly why a single task was chosen for it: on a multi-task mix ACT cannot see the instruction, so identical frames from different tasks would only teach it a cross-task average.

Task	Episodes	Frames
pick up a cube and place it in the bin	300	106,085

Chosen from a nine-task set because it has the joint-highest episode count, and a cube is a rigid body with clean geometry and stable appearance — the most standard pick-and-place benchmark of the nine. The two stacking tasks need precise alignment and are clearly harder, so they were excluded.

Data

• Source: the pick_up_a_cube_and_place_in_the_bin split of zhuzhuangtian/so101-pick-place-tasks on ModelScope (Apache-2.0), LeRobot v3.0 format.
• Robot: SO-ARM101 follower arm, 6 DoF.
• Cameras: two RGB streams, observation.images.top and observation.images.wrist, 480×640 each, 30 fps.
• observation.state and action are both float32(6): shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper (joint positions, absolute targets).

Training

Base	none — trained from scratch; vision backbone is torchvision ImageNet ResNet-18
Params	≈ 51.6 M
Architecture	4 encoder / 1 decoder layers, dim_model 512, VAE objective enabled
Steps	50,000 (≈ 11.3 epochs at effective batch 24)
Batch	8 per device × 3 devices = 24 effective
Optimizer	AdamW, lr 1e-5 constant, wd 1e-4, betas (0.9, 0.999)
Schedule	none — ACT's LeRobot preset returns no scheduler (no warmup, no decay)
Action chunk	100 (chunk_size = n_action_steps = 100)
Augmentation	brightness/contrast 0.7–1.3, hue ±0.05, saturation 0.5–1.5, sharpness 0.5–1.5, affine ±5° / translate 0.05; up to 5 of 6 sampled per frame
Normalization	state/action/visual all MEAN_STD from real dataset statistics

Final training loss 0.126. Note that ACT's loss is not comparable to that of the π0/SmolVLA siblings below — constant LR versus cosine schedules, and a different objective.

Published checkpoint = step 40,000, the best of 10 saved checkpoints on the metrics below.

Offline evaluation

No simulator exists for this rig, so evaluation is offline only, against a copy-the-current-joint-angles baseline:

• ratio = MAE(prediction, action) / MAE(current_state, action) — must be < 1.
• delta_corr = corr(prediction − state, action − state) — must be > 0.
• delta_std — ratio of predicted to true motion magnitude; ≈ 1 is healthy.

Best checkpoint (step 40,000), 200 frames sampled uniformly across the task:

	value	verdict
worst_ratio	1.681 (wrist_roll)	✗ fails the < 1 bar
best joint ratio	0.475 (gripper)	ok
worst delta_corr	0.618	direction is learned
delta_std	1.086 – 1.478, all six joints > 1	over-shoots

Failure mode: amplitude over-shoot, not wrong direction. delta_corr is respectable (0.618–0.870), so the policy has learned where to move; but delta_std > 1 on every joint means it predicts increments 9–48 % larger than ground truth, and that inflation pushes total error past the do-nothing baseline. All 10 checkpoints fail the bar (worst_ratio 1.905 → 1.688), and the curve is flat after step 20,000 — this is not under-training, so simply training longer is not the fix. Plausible directions instead: temporal ensembling of chunks at inference, a lower action chunk size, or loss/normalization changes.

The weakest axes are the two wrist DoF (wrist_roll, wrist_flex) — the same weak spot the π0 and SmolVLA models show, on data from the same source. Across three different architectures, that points at the data rather than at any one model.

Real-robot status

Not validated on hardware, and not recommended for hardware. The over-shoot failure mode is the kind that gets worse in closed loop, where errors accumulate. If you run it anyway, reduce speed and keep a hand on the e-stop.

Usage

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("Harrysunshine/so101-act-cube")
# batch: observation.state (6,), observation.images.top (3,480,640),
#        observation.images.wrist (3,480,640).  No language input.
action = policy.select_action(batch)

The two *normalizer_processor.safetensors files are required. Without them LeRobot silently falls back to identity normalization: loss looks fine, but every action comes out at the wrong scale. Load the repo as a whole, do not cherry-pick model.safetensors.

Only inference assets are published here. Optimizer/scheduler/RNG state is not included, so this repo cannot be used to resume training.

Limitations

• Does not pass the offline acceptance bar (worst_ratio 1.681 ≥ 1). Reference artifact, not a deployable policy.
• No language conditioning — one task only, cannot be re-targeted by prompt.
• Rig-specific top camera. The training data comes from a different physical setup than ours: the wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the top camera extrinsics differ. Dropping this checkpoint onto another SO-ARM101 will not work without camera adaptation or a short fine-tune on target-rig data.
• Absolute joint-position action space; sending these as deltas will diverge in closed loop.
• Fixed 480×640 dual-camera observation.
• Single cube on the training tabletop; no generalization claims beyond it.

Comparison note

This model is a single-task run; the two siblings below are 9-task mixes on a superset of the same source data. Different training sets, so the metric magnitudes cannot be ranked against each other. What is comparable is the failure mode and the location of the weak axes: ACT over-shoots (delta_std > 1) where π0 and SmolVLA under-shoot (0.71–0.99), and all three are weakest at the wrist.

• Harrysunshine/so101-pi0-9task — π0 full fine-tune, 9 tasks, passes the bar.
• Harrysunshine/so101-smolvla-9task — SmolVLA, 9 tasks, passes the bar.

License

Apache-2.0, following LeRobot and the source dataset.

---

中文说明

SO-ARM101 桌面抓放 ACT 单任务（方块放进筐）权重，从零训练（视觉塔＝torchvision ImageNet ResNet-18），约 51.6M 参数。

先看这条：本 checkpoint 没过我们的离线硬底线（worst_ratio 1.681，要求 < 1）， 作为对照与参考产物发布，不是可部署策略。

• 为什么是单任务：lerobot 的 ACT 不接语言输入。9 任务混训时它看不到任务描述， 同样画面+关节角下只能学跨任务平均动作。九任务里挑方块＝集数并列最多、刚体几何清晰外观稳定， 两个 Stack 要精确对准明显更难，排除。
• 配方＝ACT 官方默认：LR 1e-5 恒定（ACT 的 preset 不返回 scheduler，无 warmup 无衰减）， 50k 步、有效 batch 24（epoch 11.31）、action chunk 100。
• 三个模型的 loss 不可横向比：ACT 恒定 LR，π0/SmolVLA 都是余弦，只有各自收敛后的验收指标可比。
• 发布第 40,000 步，10 存点里最优。
• 失效模式＝幅度过冲，与 π0/SmolVLA 方向相反：delta_corr 0.618–0.870 其实不低（方向学对了）， 但 delta_std 六关节全 > 1（1.086–1.478）＝预测增量比真值大 9–48%，误差被放大到超过照抄基线。 10 个点全不过门、20k 步后曲线就平了 ⇒ 不是训练不足，再训不解决；可试的方向是推理端 chunk 时序集成、 减小 chunk、或改损失/归一化口径。
• 腕部两自由度最差，与 π0/SmolVLA 一致 ⇒ 跨三个架构同一短板，是数据的难点。
• 真机未验证、也不建议上真机：过冲这种失效模式在闭环里会累积。真要跑就降速 + 手放急停。
• 两个 *normalizer_processor.safetensors 必须一起加载，缺了静默退回恒等归一化、动作尺度整体错。
• 边界：top 相机外参与我们自己的臂不同，换机器需相机适配或少量目标机器数据再微调。