--- license: apache-2.0 tags: - robotics - lerobot - act - so101 - so-arm101 - manipulation - pick-and-place pipeline_tag: robotics library_name: lerobot --- # SO-ARM101 pick-and-place — ACT, single task (cube) An [ACT](https://tonyzhaozh.github.io/aloha/) (Action Chunking Transformer) policy trained from scratch on one SO-ARM101 tabletop task: pick up a cube and place it in the bin. > **Read this first.** This checkpoint **does not clear our offline acceptance bar** — on every one of > its six joints, its best step still predicts motion with larger error amplitude than a > "hold current pose" baseline on at least one axis (`worst_ratio` 1.681, needs < 1). It is published > as a **reference and comparison artifact, not a deployable policy.** See > [Offline evaluation](#offline-evaluation). ## Task Single task, no language conditioning. LeRobot's ACT implementation **takes no language input** — which is exactly why a single task was chosen for it: on a multi-task mix ACT cannot see the instruction, so identical frames from different tasks would only teach it a cross-task average. | Task | Episodes | Frames | |---|---|---| | pick up a cube and place it in the bin | 300 | 106,085 | Chosen from a nine-task set because it has the joint-highest episode count, and a cube is a rigid body with clean geometry and stable appearance — the most standard pick-and-place benchmark of the nine. The two stacking tasks need precise alignment and are clearly harder, so they were excluded. ## Data - Source: the `pick_up_a_cube_and_place_in_the_bin` split of [`zhuzhuangtian/so101-pick-place-tasks`](https://www.modelscope.cn/datasets/zhuzhuangtian/so101-pick-place-tasks) on ModelScope (Apache-2.0), LeRobot v3.0 format. - Robot: SO-ARM101 follower arm, 6 DoF. - Cameras: two RGB streams, `observation.images.top` and `observation.images.wrist`, 480×640 each, 30 fps. - `observation.state` and `action` are both `float32(6)`: `shoulder_pan`, `shoulder_lift`, `elbow_flex`, `wrist_flex`, `wrist_roll`, `gripper` (joint positions, **absolute** targets). ## Training | | | |---|---| | Base | none — trained from scratch; vision backbone is torchvision ImageNet ResNet-18 | | Params | ≈ 51.6 M | | Architecture | 4 encoder / 1 decoder layers, `dim_model` 512, VAE objective enabled | | Steps | 50,000 (≈ 11.3 epochs at effective batch 24) | | Batch | 8 per device × 3 devices = 24 effective | | Optimizer | AdamW, lr **1e-5 constant**, wd 1e-4, betas (0.9, 0.999) | | Schedule | none — ACT's LeRobot preset returns no scheduler (no warmup, no decay) | | Action chunk | 100 (`chunk_size` = `n_action_steps` = 100) | | Augmentation | brightness/contrast 0.7–1.3, hue ±0.05, saturation 0.5–1.5, sharpness 0.5–1.5, affine ±5° / translate 0.05; up to 5 of 6 sampled per frame | | Normalization | state/action/visual all MEAN_STD from real dataset statistics | Final training loss 0.126. Note that ACT's loss is **not comparable** to that of the π0/SmolVLA siblings below — constant LR versus cosine schedules, and a different objective. **Published checkpoint = step 40,000**, the best of 10 saved checkpoints on the metrics below. ## Offline evaluation No simulator exists for this rig, so evaluation is offline only, against a **copy-the-current-joint-angles baseline**: - `ratio = MAE(prediction, action) / MAE(current_state, action)` — must be **< 1**. - `delta_corr = corr(prediction − state, action − state)` — must be **> 0**. - `delta_std` — ratio of predicted to true motion magnitude; ≈ 1 is healthy. Best checkpoint (step 40,000), 200 frames sampled uniformly across the task: | | value | verdict | |---|---|---| | worst_ratio | **1.681** (`wrist_roll`) | ✗ fails the < 1 bar | | best joint ratio | 0.475 (`gripper`) | ok | | worst delta_corr | 0.618 | direction is learned | | delta_std | **1.086 – 1.478, all six joints > 1** | over-shoots | **Failure mode: amplitude over-shoot, not wrong direction.** `delta_corr` is respectable (0.618–0.870), so the policy has learned *where* to move; but `delta_std > 1` on every joint means it predicts increments 9–48 % larger than ground truth, and that inflation pushes total error past the do-nothing baseline. All 10 checkpoints fail the bar (`worst_ratio` 1.905 → 1.688), and the curve is flat after step 20,000 — **this is not under-training**, so simply training longer is not the fix. Plausible directions instead: temporal ensembling of chunks at inference, a lower action chunk size, or loss/normalization changes. **The weakest axes are the two wrist DoF** (`wrist_roll`, `wrist_flex`) — the same weak spot the π0 and SmolVLA models show, on data from the same source. Across three different architectures, that points at the data rather than at any one model. ## Real-robot status **Not validated on hardware, and not recommended for hardware.** The over-shoot failure mode is the kind that gets worse in closed loop, where errors accumulate. If you run it anyway, reduce speed and keep a hand on the e-stop. ## Usage ```python from lerobot.policies.act.modeling_act import ACTPolicy policy = ACTPolicy.from_pretrained("Harrysunshine/so101-act-cube") # batch: observation.state (6,), observation.images.top (3,480,640), # observation.images.wrist (3,480,640). No language input. action = policy.select_action(batch) ``` The two `*normalizer_processor.safetensors` files are **required**. Without them LeRobot silently falls back to identity normalization: loss looks fine, but every action comes out at the wrong scale. Load the repo as a whole, do not cherry-pick `model.safetensors`. Only inference assets are published here. Optimizer/scheduler/RNG state is not included, so this repo cannot be used to resume training. ## Limitations - **Does not pass the offline acceptance bar** (`worst_ratio` 1.681 ≥ 1). Reference artifact, not a deployable policy. - No language conditioning — one task only, cannot be re-targeted by prompt. - **Rig-specific top camera.** The training data comes from a different physical setup than ours: the wrist view matches pixel-for-pixel (same SO-ARM101 follower hardware), but the **top camera extrinsics differ**. Dropping this checkpoint onto another SO-ARM101 will not work without camera adaptation or a short fine-tune on target-rig data. - Absolute joint-position action space; sending these as deltas will diverge in closed loop. - Fixed 480×640 dual-camera observation. - Single cube on the training tabletop; no generalization claims beyond it. ## Comparison note This model is a **single-task** run; the two siblings below are **9-task** mixes on a superset of the same source data. Different training sets, so **the metric magnitudes cannot be ranked against each other.** What is comparable is the failure mode and the location of the weak axes: ACT over-shoots (`delta_std` > 1) where π0 and SmolVLA under-shoot (0.71–0.99), and all three are weakest at the wrist. - [`Harrysunshine/so101-pi0-9task`](https://huggingface.co/Harrysunshine/so101-pi0-9task) — π0 full fine-tune, 9 tasks, passes the bar. - [`Harrysunshine/so101-smolvla-9task`](https://huggingface.co/Harrysunshine/so101-smolvla-9task) — SmolVLA, 9 tasks, passes the bar. ## License Apache-2.0, following LeRobot and the source dataset. --- ## 中文说明 SO-ARM101 桌面抓放 **ACT 单任务**（方块放进筐）权重，从零训练（视觉塔＝torchvision ImageNet ResNet-18），约 51.6M 参数。 > **先看这条**：本 checkpoint **没过我们的离线硬底线**（`worst_ratio` 1.681，要求 < 1）， > 作为**对照与参考产物**发布，**不是可部署策略**。 - **为什么是单任务**：lerobot 的 ACT **不接语言输入**。9 任务混训时它看不到任务描述， 同样画面+关节角下只能学跨任务平均动作。九任务里挑方块＝集数并列最多、刚体几何清晰外观稳定， 两个 Stack 要精确对准明显更难，排除。 - 配方＝ACT 官方默认：**LR 1e-5 恒定**（ACT 的 preset 不返回 scheduler，无 warmup 无衰减）， 50k 步、有效 batch 24（epoch 11.31）、action chunk 100。 - **三个模型的 loss 不可横向比**：ACT 恒定 LR，π0/SmolVLA 都是余弦，只有各自收敛后的验收指标可比。 - **发布第 40,000 步**，10 存点里最优。 - **失效模式＝幅度过冲**，与 π0/SmolVLA 方向相反：`delta_corr` 0.618–0.870 其实不低（方向学对了）， 但 `delta_std` **六关节全 > 1**（1.086–1.478）＝预测增量比真值大 9–48%，误差被放大到超过照抄基线。 10 个点全不过门、**20k 步后曲线就平了 ⇒ 不是训练不足**，再训不解决；可试的方向是推理端 chunk 时序集成、 减小 chunk、或改损失/归一化口径。 - **腕部两自由度最差**，与 π0/SmolVLA 一致 ⇒ 跨三个架构同一短板，是数据的难点。 - **真机未验证、也不建议上真机**：过冲这种失效模式在闭环里会累积。真要跑就降速 + 手放急停。 - 两个 `*normalizer_processor.safetensors` 必须一起加载，缺了静默退回恒等归一化、动作尺度整体错。 - **边界**：top 相机外参与我们自己的臂不同，换机器需相机适配或少量目标机器数据再微调。